Open bene-starzengruber opened 10 months ago
jennifer-shehane
commented
10 months ago We made an update from webpack 4 to webpack 5 in 12.17.4, I wonder if it has anything to do with that upgrade. https://docs.cypress.io/guides/references/changelog#12-17-4
cubanx
commented
10 months ago We are getting the same error also intermittently. I don't think we had it as far back as 12.17.4 but I can't say that for certain.
It only seems to happen toward the end of our Component Test runs. We have 1500+ component tests that we are running.
The following error originated from your test code, not from Cypress.
> Loading chunk cypress-support-file failed.
(error: http://localhost:8080/__cypress/src/cypress-support-file.js)
When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.
Cypress could not associate this error to any specific test.
We dynamically generated a new test to display this failure.
ChunkLoadError: The following error originated from your test code, not from Cypress.
> Loading chunk cypress-support-file failed.
(error: http://localhost:8080/__cypress/src/cypress-support-file.js
When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.
Cypress could not associate this error to any specific test.
We dynamically generated a new test to display this failure.
at __webpack_require__.f.j (http://localhost:8080/__cypress/src/main.js:8702:29
at <unknown> (http://localhost:8080/__cypress/src/main.js:8552:40
at Array.reduce (<anonymous>)
at __webpack_require__.e (http://localhost:8080/__cypress/src/main.js:8551:67
at Object.load (http://localhost:8080/__cypress/src/main.js:1510:582
at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:110819:21
at tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1807:23
at Object.gotValue (http://localhost:8080/__cypress/runner/cypress_runner.js:6476:18
at Object.gotAccum (http://localhost:8080/__cypress/runner/cypress_runner.js:6465:25
at Object.tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1807:23
[Open in build log](https://teamcity.haivision.com/buildConfiguration/CineNet4x_Projects_CypressComponents/690401?hideProblemsFromDependencies=false&expandBuildTestsSection=true&hideTestsFromDependencies=false&expandBuildChangesSection=true&showLog=690401_2000000247_2000000247&logFilter=debug)
werge2121
commented
10 months ago Same for us too
amir1218
commented
10 months ago We have been running into this sporadically when Component testing after migrating from CY 12.x to 13.6.2. Similarly to cases above, it happens towards the end of running a batch of suites (files) together.
Resorted to running cypress with DEBUG='cypress:*'. This is an instance of a crash:
cypress: server: reporter got mocha event 'test'
with args: [u {
title: 'An uncaught error was detected outside of a test',
fn: [Function: o] {
toString: [Function(anonymous)]
},
body: '() => {\n throw err;\n }',
async: undefined,
sync: undefined,
_timeout: 2000,
_slow: 250,
_enableTimeouts: true,
timedOut: undefined,
_retries: -1,
_currentRetry: 0,
pending: false,
type: 'test',
duration: undefined,
state: 'skipped',
parent: p {
title: '',
ctx: {},
suites: [],
tests: [Array],
pending: false,
_beforeEach: [],
_beforeAll: [],
_afterEach: [],
_afterAll: [],
root: true,
_timeout: 2000,
_enableTimeouts: true,
_slow: 250,
_bail: false,
_retries: -1,
_onlyTests: [],
_onlySuites: [],
delayed: false,
_events: [Object: null prototype],
_eventsCount: 1,
file: 'VALID_PATH_TO_FILE_DEDACTED',
id: 'r1',
type: 'suite'
},
id: 'r2',
_testConfig: {
testConfigList: [],
unverifiedTestConfig: {}
},
order: 1,
invocationDetails: {
function: 'Object.getInvocationDetails',
fileUrl: 'http://localhost:8080/__cypress/runner/cypress_runner.js',
originalFile: 'http://localhost:8080/__cypress/runner/cypress_runner.js',
line: 97377,
column: 17,
whitespace: ' ',
stack: 'Error\n' + ' at Object.getInvocationDetails
(http://localhost:8080/__cypress/runner/cypress_runner.js:97377:17)\n' + ' at Suite.addTest
(http://localhost:8080/__cypress/runner/cypress_runner.js:145737:85)\n' + ' at Object.createRootTest
(http://localhost:8080/__cypress/runner/cypress_runner.js:145872:21)\n' + ' at Object.normalizeAll
(http://localhost:8080/__cypress/runner/cypress_runner.js:163115:17)\n' + ' at CDPBrowserSocket.<anonymous>
(http://localhost:8080/__/assets/index-caf2649c.js:146264:46)\n' + ' at CDPBrowserSocket.on2
(http://localhost:8080/__/assets/index-caf2649c.js:34575:11)\n' + ' at Emitter2.emit
(http://localhost:8080/__/assets/index-caf2649c.js:34616:24)\n' + ' at http://localhost:8080/__/assets/index-caf2649c.js:37709:15'
}
}] + 9 msThere is a lot in the logs that makes it hard to share but please do let us know if we should be looking for or can share anything useful to help unraveling this.
cubanx
commented
10 months ago Does anyone know of a way to suppress this error until we can get a resolution?
rbecheras
commented
9 months ago We have a very similar issue with another file: node_modules_core-js_modules_es_array_concat_js-node_modules_babel_runtime_helpers_esm_define-ec4858
1) An uncaught error was detected outside of a test
0 passing (433ms)
1 failing
1) An uncaught error was detected outside of a test:
ChunkLoadError: The following error originated from your test code, not from Cypress.
> Loading chunk node_modules_core-js_modules_es_array_concat_js-node_modules_babel_runtime_helpers_esm_define-ec4858 failed.
(error: http://localhost:8080/__cypress/src/node_modules_core-js_modules_es_array_concat_js-node_modules_babel_runtime_helpers_esm_define-ec4858.js)
When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.
Cypress could not associate this error to any specific test.It makes our test suite pass sometimes but fail randomly and frequently, on a component or another, randomly.
benpatterson
commented
9 months ago I'm leaving behind some notes after digging into this further.
We are experiencing this error on ~9% of our component test builds. (I can't remember offhand how many tests that entails.)
Indeed error messages above are consistent with what we see, here's an excerpt (with a slight edit on chunk name tbh):
1) An uncaught error was detected outside of a test:
ChunkLoadError: The following error originated from your test code, not from Cypress.
> Loading chunk name1-name2-file~spec-11~spec-14~spec-15~spec-17~spec-22~spec-23~spec-25~spec-27~spec-28~43d59dbf failed.It includes the localhost path as well on the next line (which I left out here).
Noteworthy...going up a few lines in the build, we are seeing something like this (I changed more file names out of caution):
Running: dir/myComponentSpec.spec.tsx
Estimated: 1 second
ℹ ｢wdm｣: Compiling...
(Attempt 1 of 3) An uncaught error was detected outside of a test
ℹ ｢wdm｣: wait until bundle finished: /<root+directory>/related-file~spec-34.js
ℹ ｢wdm｣: wait until bundle finished: /<root+directory>/related-file.js
(Attempt 2 of 3) An uncaught error was detected outside of a test
1) An uncaught error was detected outside of a testIt appears to me that the ChunkLoadError occurs because the next test is running before webpack compilation is completed on the devserver. I think this includes related CSS compilations, but honestly I don't know if that's all the time or a subset of examples.
Ultimately I'm convinced the next test needs to wait until compilation completes and the devserver has updated the cache. I attempted to set devserver settings around hot reload, etc, but those didn't work -- likely because the test is still trying to serve the page and is not waiting for the devserver to signal it's up to date.
When it comes to waiting for compilation to complete, we do have logic around this in CypressCTWebpackPlugin in this function. As-written, it's awaiting specs, and this is why I'm wondering if we need something to wait for related assets, too. Regardless, I think we need a way of ensuring all compilation completes, including any styling, before proceeding to the next test.
There had been a previous PR that leveraged the webpack devserver's index HTML file, which I believe would yield the result I'm thinking of (change here). That PR was intended to fix a situation specific to MacOS version, but I'm optimistic it could resolve the ChunkLoadError. The PR was merged but the code in the file today is older than the PR, so I don't know if it was rolled back at a later time.
hogatejon
commented
9 months ago Just commenting to say I am also experiencing this issue. Have traced it down to being introduced by 12.17.4, any version below this seems to be completely fine on our CI runs.
jennifer-shehane
commented
9 months ago jennifer-shehane
commented
9 months ago We're spiking into this to see if we can find what's causing this.
TomaszG
commented
9 months ago Same here. Since the v12 upgrade (I don't recall what the version was, though), we have gotten this on GitHub Actions CI sometimes. My initial thought was that it was caused by the first spec in our test suite. However, changing the order of suites didn't help. It's just that the first spec fails.
My workaround for this was to amend the run script:
cypress run --component --spec './**/OurFirstSpec.cy.js' || true && cypress run --component
So, the first attempt could fail, but the second run was 100% successful. Since upgrading from 13.6.3 to 13.6.4, the second run also fails for around 30% of runs.
Cypress 13.6.4 Node.js 21.6.1 NextJS 14.1.0
BTW, this seems to be related or even the same: https://github.com/cypress-io/cypress/issues/16421
AtofStryker
commented
9 months ago Hey all. Sorry for the delay on an update here. I've been investigating the issue heads down for about a week now. I'm able to reproduce the error on my end but only at the start of the run and sparingly in the middle of the run. I think there are two issues possibly going on here
At the end of webpack compilation, the dev-server:compile:success event is emitted by the cypress webpack dev server, but nothing is actively listening to this event. The only exception is watchForChanges being enabled inside the SocketCT handler inside the Cypress server, which is unset in run mode and isn't applicable here.
In other words, we can get ourselves into a race condition where webpack takes a long time to build all assets, but the cypress run has begun and is able to resolve the index.html chunk, but not other chunks in the test because they are not present in the output directory. That is what is happening in this example:
Adding to what @benpatterson mentioned and a bit more complicated to reproduce, but does happen, is webpack-dev-server recompiling in the middle of a test. We can see something like this here with some logging I added in this diagnostic binary.
webpack-dev-server to recompile. test fails with chunk errorFor this, it seems like the dev-server is watching assets and recompiling. What I need to further investigate is
webpack.config from version 4 to 5.Waiting for the dev server to compile is a bit tricky as it can trigger a compilation at any time, but if we can answer the above question, it puts us in a better position to solve the problem correctly.
CYPRESS_INTERNAL_BROWSER_CONNECT_TIMEOUT=120000 DEBUG='cypress:server:open_project,cypress:webpack-dev-server:CypressCTWebpackPlugin npx cypress run --component ... which might give us an idea what webpack is doing under your test.I will give more updates as soon as I have them. I appreciate everyones patience with the issue resolution.
TomaszG
commented
9 months ago @AtofStryker, great investigation and summary. I can confirm that the error happens only at the beginning of the run for me; I haven't noticed it in the middle. I also haven't noticed any webpack re-compilation in the middle of the run.
The compilation times on CI are around 19 seconds for the first run (which is allowed to fail) and 14 seconds for the second run.
AtofStryker
commented
9 months ago @bene-starzengruber @cubanx @werge2121 @amir1218 @rbecheras @TomaszG what version of webpack-dev-server are you all using? It should be pretty easy to tell with either in your package.json or from debugging with cypress via DEBUG=cypress:webpack-dev-server:devServer npx cypress run --component ...
I think the recompiling in the middle is related to either a bug or a misconfiguration within webpack-dev-server@3 . The documentation is difficult to find since v3 hasn't been released or maintained for about 3 years, but from what I can see everything looks configured correctly on our end. Updating to webpack-dev-server@4 seems to resolve the issue of the mid run recompilation. Cypress ships with webpack-dev-server@4 with @cypress/webpack-dev-server unless you explicitly install v3
@TomaszG I would be curious if using the diagnostic binary while setting CYPRESS_INTERNAL_BROWSER_CONNECT_TIMEOUT=120000 resolves your issue of the initial chunk load error failure if you have a chance to use it.
werge2121
commented
9 months ago We should just be using the default devserver that comes with cypress, so 4. For us its happening in the middle of the run usually. It takes around 5-6 seconds for the webpack to compile usually.
cwillsea-mtg
commented
9 months ago My team is using the default dev server that comes with cypress:
component: {
devServer: {
bundler: 'webpack',
framework: 'next',
},
},The errors seem to always happen at the start of a test file, but pop up in random places in our test suite. For example, here is a recent failure in the 73th test file out of a suite of 90 where the other 89 runs all passed (edited to remove component/file names):
Running: components/COMPONENT_NAME.cy.tsx (73 of 90)
1) An uncaught error was detected outside of a test
0 passing (665ms)
1 failing
1) An uncaught error was detected outside of a test:
ChunkLoadError: The following error originated from your test code, not from Cypress.
> Loading chunk vendors-node_modules_SOME_STRING.js failed.
(error: http://localhost:8080/__cypress/src/vendors-SOME_STRING.js)
When Cypress detects uncaught errors originating from your test code it will automatically fail the current test.
Cypress could not associate this error to any specific test.
We dynamically generated a new test to display this failure.
at __webpack_require__.f.j (http://localhost:8080/__cypress/src/webpack.js:340:29)
at <unknown> (http://localhost:8080/__cypress/src/webpack.js:137:40)
at Array.reduce (<anonymous>)
at __webpack_require__.e (http://localhost:8080/__cypress/src/webpack.js:136:67)
at Object.load (http://localhost:8080/__cypress/src/main.js:485:577)
at <unknown> (http://localhost:8080/__cypress/runner/cypress_runner.js:110819:21)
at tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1807:23)
at Object.gotValue (http://localhost:8080/__cypress/runner/cypress_runner.js:6476:18)
at Object.gotAccum (http://localhost:8080/__cypress/runner/cypress_runner.js:6465:25)
at Object.tryCatcher (http://localhost:8080/__cypress/runner/cypress_runner.js:1807:23)
(Results)
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Tests: 1 │
│ Passing: 0 │
│ Failing: 1 │
│ Pending: 0 │
│ Skipped: 0 │
│ Screenshots: 1 │
│ Video: false │
│ Duration: 0 seconds │
│ Spec Ran: components/COMPONENT_NAME.cy.tsx │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
The frequency with which these errors occur has been increasing as we add more test files. If there's any debug/diagnostic config we can set to get more info happy to try as this issue is happening frequently in our CI
anilanar
commented
8 months ago @AtofStryker
It would be great if cypress team would develop a pre-bundle option for runs in the CI. We had similar issues with MSW in our component tests. Browser stops trying when a service worker script takes more than ~60 seconds to load. We had to slightly modify the default dev server to "prebuild" service worker file so that when it's actually needed, it's already built and is ready to serve by the dev server:
devServer: async ({ specs, cypressConfig, devServerEvents }) => {
const result = await devServer({
specs,
cypressConfig,
devServerEvents,
webpackConfig,
});
// trigger a build for MSW so that it doesn't time out
// in CI when trying to initialize msw
await fetch('http://localhost:8080/mockServiceWorker.js');
return result;
},So if there were an option to pre-bundle everything, flaky tests due to middle of the run builds taking too long would be long gone.
In user land, this can be achieved by manually pre-bundling and then by creating a custom dev-server that serves pre-bundled files. But it's not very elegant, Cypress team can do better by introducing a new config for pre-bundling.
AtofStryker
commented
8 months ago Hey all. Just a few updates.
Mid recompilation with webpack-dev-server has stopped with the update from v3 to v4, which is great! However, we are still hitting ChunkLoadErrors 🙁 .
This issue has been extremely challenging to reproduce, but we have a few theories. My guess is that there are chunk loading issues associated with the main bundle mainly due to us bundling every component test, regardless of what is used under test. The only exception to this is if the --spec option is leveraged to pass in specs. For example:
cypress open is used to to develop a single component test and there are 39 other component tests, ALL 40 component tests are compiled when the app is launched.cypress run is used to test all specs, ALL component tests are compiled before the tests are initialized. cypress run is used to test multiple specs via the --parallel flag in CI, ALL component tests are compiled before the tests are initialized, regardless of what tests are run.cypress run is used to test a single spec via the spec argument, ONE single spec is compiled before the test is run.cypress run is used to test multiple specs via the spec argument, ONLY THE SPECS PASSED IN are compiled before the test is run.The larger N is in this case (our example above was 40), the worse the dev server scales as far as building is concerned. Each spec is considered its own entry point in webpack. We don't have proof that reducing the bundle size would fix the chunk errors, but I think this is in large part of the problem as this seems to happen mostly in projects where there are dozens of CT tests.
In most cases, even if the build takes minutes, cypress waits for the dev server to return a response before executing tests. @anilanar In other words, we are pre-bundling everything needed under test already but the assets are not loaded into the browser all at once, which might explain the service worker issue you are running into and I'm glad you were able to find a solution! Something is causing the chunk to fail to load meaning it either doesn't exist, or it timed out loading. Since the assets are held in memory, it is difficult to figure out which has occurred. Either way, I am not convinced flaky tests are due to builds taking too long.
What we are trying internally with one of our projects where this error happens frequently, is to leverage the --spec option to group our tests under execution and use circleci to split the tests per container (@cwillsea-mtg this might be something you can try depending on your setup and would be great feedback for us if it lessens or resolve the issue for you). This will only bundle the assets needed under test, reducing the chunk and bundle sizes under test and likely the stress to load them into memory. This could give us direction on if we need to change some aspects of the dev server depending on whether ChunkLoadErrors continue to surface or not and our ability to tune them. I will report back when I gather more data/information.
cubanx
commented
8 months ago @AtofStryker We are using whatever webpack-dev-server comes with Cypress, so I assume it is version 4.
I concur that it seems large tests suites seem to cause this issue. To both speed up our test runs and reduce the frequency of this error, I have some PowerShell scripts that use the --spec parameter to split up the test suite and run a subset of our component tests in parallel.
This reduced the number of test per run from ~1500 to ~400 per run, and the error seems to have abated.
If this helps at all, even running interactively, if we load all our Component tests and then try to run them, there is a significant delay between update/reloads or if you switch test files.
I use another PowerShell script to modify the specPattern inside the cypress.config.ts to limit the number of tests I'm running interactively to just a handful. It's helpful both for speed ups of test reloading and to reduce the test clutter since we have so many.
We also tried switching the Component Server in the past to vite. The interactive tests became too slow to use. We are currently running in a bifurcated fashion, with the full up dev server running vite, but the Component Tests running webpack.
I am actually about to leave the current position I'm in, but if there's any diagnostics or info that would help you, I'll be here until March 1st.
Thanks for your work on this!
anilanar
commented
8 months ago @AtofStryker Webpack supports lazy loading with async chunks (code splitting) therefore it is definitely possible to make a test suite start downloading and bundling stuff in the middle of the run. Webpack dev server doesn't initiate building an async chunk unless that chunk is requested by the client with an async import.
This issue can be solved with pre-bundling. It can probably be solved without pre-bundling too, if there is a way for webpack dev server to report any async chunks it finds during a build. Then those async chunks can be requested manually (like I did with the service worker) to manually trigger a build for them before running a test. Keep in mind that this needs to be done recursively, until no async chunks are left.
Another idea is to disable code splitting, e.g. https://webpack.js.org/plugins/limit-chunk-count-plugin/
I'm not really sure if this has anything to do with your ChunkLoadErrors though. What I know is, if an async import fails, it will reject with a ChunkLoadError.
A simple test that downloads an async chunk in the middle of the run:
export const Example = () => {
const [done, setDone] = useState(false);
return (
<button
onClick={async () => {
await import("./path-to-something-very-slow-to-bundle");
setDone(true);
}}
>
{done ? "DONE" : "Click me to download async chunk"}
</button>
);
};
it('will download async chunk', () => {
cy.mount(<Example />);
cy.get('button').click();
cy.contains('DONE').should('be.visible');
});Edit: I still think it's better to pre-bundle and run cypress without a dev server and without a file watcher and without no recompiles for CI use cases; instead of trying to fight file watcher/unnecessary rebuilds that can be caused by myriads of webpack plugins (and potentially unintentional bugs in them). I think using dev-server on CI is picking a wrong fight. Consider this: no build tool or no test tool runs in watch mode on CI. So cypress is opening a can of worms by running in watch mode.
amir1218
commented
8 months ago @AtofStryker We are on webpack-dev-server 4.15.1
werge2121
commented
8 months ago Any updates?
anilanar
commented
8 months ago Try this workaround, as I previously suggested:
// cypress.config.js
const { defineConfig } = require('cypress');
const { devServer } = require('@cypress/webpack-dev-server');
module.exports = defineConfig({
...
devServer: async ({ specs, cypressConfig, devServerEvents }) => {
const result = await devServer({
specs,
cypressConfig,
devServerEvents,
webpackConfig,
});
// prebuild files that can cause test failures when they take too long
// during runs
await Promise.all([
fetch('http://localhost:8080/__cypress/src/cypress-support-file.js'),
// prebuild any other file, such as service workers here
]);
return result;
},
});If other async chunks are causing middle-of-the-run builds, then try using https://webpack.js.org/plugins/limit-chunk-count-plugin/ and limit your chunk count to 1. Or try turning off chunk splitting, see https://webpack.js.org/plugins/split-chunks-plugin/#optimizationsplitchunks .
Again, I doubt this is a misconfiguration in webpack dev server. Async chunks are normal things that optimize initial bundle size by lazy loading things later. They are usually caused by async imports, but if you don't have an async import on your side or if you don't know what that even means, then it must be used by one of the libraries you use.
This is a problem with async chunks triggering builds and those builds not finishing before test times out.
The solution is one of:
Edit: typo
TomaszG
commented
8 months ago @AtofStryker I've tried what you've recommended, so I've set CYPRESS_INTERNAL_BROWSER_CONNECT_TIMEOUT=120000 and also used your diagnostic binary based on 13.6.5 and after a couple of runs, it seems that there are no failures. Here's the log output from the tests:
cypress:webpack-dev-server:start using webpack-dev-server v4 +0ms
cypress:webpack-dev-server:CypressCTWebpackPlugin beforeCompile +0ms
cypress:webpack-dev-server:CypressCTWebpackPlugin beforeCompile projectRoot: /home/runner/work/<REDACTED>/apps/client +0ms
cypress:webpack-dev-server:CypressCTWebpackPlugin beforeCompile supportFile: /home/runner/work/<REDACTED>/apps/client/cypress/support/component.js +0ms
cypress:webpack-dev-server:CypressCTWebpackPlugin beforeCompile indexHtmlFile: cypress/support/component-index.html +0ms
cypress:webpack-dev-server:CypressCTWebpackPlugin beforeCompile: compile hooks not registered. Invoking callback. +0ms
cypress:webpack-dev-server:CypressCTWebpackPlugin addCompilationHooks +45ms
cypress:webpack-dev-server:CypressCTWebpackPlugin addCompilationHooks: Webpack 5 detected +5ms
cypress:webpack-dev-server:CypressCTWebpackPlugin beforeCompile +25ms
cypress:webpack-dev-server:CypressCTWebpackPlugin beforeCompile projectRoot: /home/runner/work/<REDACTED>/apps/client +14ms
cypress:webpack-dev-server:CypressCTWebpackPlugin beforeCompile supportFile: /home/runner/work/<REDACTED>/apps/client/cypress/support/component.js +0ms
cypress:webpack-dev-server:CypressCTWebpackPlugin beforeCompile indexHtmlFile: cypress/support/component-index.html +0ms
<i> [webpack-dev-server] Project is running at:
<i> [webpack-dev-server] Loopback: http://127.0.0.1:8080/
<i> [webpack-dev-server] Content not from webpack is served from '/home/runner/work/<REDACTED>/apps/client/public' directory
cypress:webpack-dev-server:devServer Component testing webpack server 4 started on port 8080 +0ms
cypress:webpack-dev-server:CypressCTWebpackPlugin beforeCompile: compile hooks registered, filtering out files that have been removed by the file system but not yet detected by the onSpecsChange handler +53ms
cypress:webpack-dev-server:CypressCTWebpackPlugin invoking callback +0ms
cypress:webpack-dev-server:CypressCTWebpackPlugin addCompilationHooks +7ms
cypress:webpack-dev-server:CypressCTWebpackPlugin addCompilationHooks: Webpack 5 detected +0ms
We detected that you have versions of dependencies that are not officially supported:
- `next`. Expected ^10.0.0 || ^11.0.0 || ^12.0.0 || ^13.0.0, found 14.1.0.
If you're experiencing problems, downgrade dependencies and restart Cypress.
cypress:webpack-dev-server:webpack projectRoot: /home/runner/work/<REDACTED>/apps/client, files: <REDACTED> +0ms
====================================================================================================
(Run Starting)
tput: No value for $TERM and no -T specified
┌────────────────────────────────────────────────────────────────────────────────────────────────┐
│ Cypress: 13.6.5 │
│ Browser: Electron 114 (headless) │
│ Node Version: v21.6.2 (/opt/hostedtoolcache/node/21.6.2/x64/bin/node) │
│ Specs: 34 found (<REDACTED>) │
│ Searched: **/*.cy.{js,jsx,ts,tsx} │
│ Experiments: experimentalWebKitSupport=true │
└────────────────────────────────────────────────────────────────────────────────────────────────┘
────────────────────────────────────────────────────────────────────────────────────────────────────
Running: <REDACTED>.cy.js (1 of 34)
cypress:webpack-dev-server:CypressCTWebpackPlugin compiler hooks done. emitting "dev-server:compile:success" to start the runner +37s
98 assets
5238 modules
client (webpack 5.86.0) compiled successfully in 37206 msI wonder if it's OK that compiler hooks done is logged after the run starts...
AtofStryker
commented
8 months ago Sorry for the delay in an update. We are continuing to experiment and try a few things on our end. @anilanar's comment of
This is a problem with async chunks triggering builds and those builds not finishing before test times out.
I think is likely the correct speculation. We are also wondering if the async chunks are being rebuilt on the fly in the dev-server cache and that is also causing more problems.
Either way, on our end we are experimenting with turning async chunking off and opting for initial chunking. The issue is @cypress/webpack-dev-server overrides this configuration for users (likely when it shouldn't), so we needed to patch the package in order to try this.
We are gathering more data with this change. It has caused a spike in some of our resource usage, but I think we have that ironed out. Once we have more information on whether the configuration changes help alleviate the symptoms I will report back!
papatelst
commented
8 months ago @AtofStryker We are getting following random error on Teamcity Docker cypress base image based UI test runs mostly during after each hook execution . We use "cypress": "13.5.1".
Is the error in stacktrace below related to this ticket as it does not say 'cypress-support-file' anywhere ?
The following error originated from your application code, not from Cypress. It was caused by an unhandled promise rejection.
> Loading chunk 59 failed.
(timeout: https://env.companyname.com/scripts/build/desktop/59.9dcc63fc.bundle.js)
When Cypress detects uncaught errors originating from your application it will automatically fail the current test.
This behavior is configurable, and you can choose to turn this off by listening to the `uncaught:exception` event.
https://on.cypress.io/uncaught-exception-from-application
Because this error occurred during a `after each` hook we are skipping all of the remaining tests.Will this issue get resolved when this ticket is resolved ?
AtofStryker
commented
8 months ago Will this issue get resolved when this ticket is resolved ?
@papatelst I think that is likely, depending on how we solve the issue. We are experimenting on our end with turning async splitChunks off, and if that experiment returns positive results, we need to adapt the @cypress/webpack-dev-server to provide defaults and not overrides to the webpack config so users are able to configure this behavior.
carbonid1
commented
8 months ago Hey everyone! Do you think there might be a way to mute this error temporarily?
I tried to handle it in cypress/support/setup.ts but had no luck as it doesn't seem to catch errors outside:
Cypress.on('fail', (err) => {
if (err.message.includes('ChunkLoadError')) {
return false
}
throw err
})Hey everyone! Do you think there might be a way to mute this error temporarily?
I tried to handle it in
cypress/support/setup.tsbut had no luck as it doesn't seem to catch errors outside:Cypress.on('fail', (err) => { if (err.message.includes('ChunkLoadError')) { return false } throw err })
@carbonid1 unfortunately I don't think that will work since the test is missing assets and failing execution. In some cases, the chunk that is failing to load (i.e. the support file) might actually contain this code.
jamestalton
commented
8 months ago We have been running into this for a while now in our component tests. Specifically with the monaco editor loading web workers.
Loading chunk vendors-node_modules_yaml_browser_index_js failed.
Our fix is to pin to Cypress version 13.5.1.
Cypress 13.5.1 works consistently. Upgrading to new versions after that, up to version 13.6.6 give us random failures in our component tests.
Looking forward to a fix.
Update: Still seeing the issue sometimes on version 13.5.1
airhorns
commented
8 months ago Ah ha -- we're also seeing the error and using monaco! We're using it via @monaco-react/loader which loads the webworkers (and the majority of the code) from a CDN, so it doesn't get built via the webpack dev server by default. Not sure if monaco itself might be the issue, or if it's just super heavyweight and triggers a race in webpack.
mirobo
commented
8 months ago Ah ha -- we're also seeing the error and using monaco! We're using it via
@monaco-react/loaderwhich loads the webworkers (and the majority of the code) from a CDN, so it doesn't get built via the webpack dev server by default. Not sure if monaco itself might be the issue, or if it's just super heavyweight and triggers a race in webpack.
We don't use React nor "monaco" (we use Angular) and we have the same issue with component tests with the ChunkLoadError.. 🤔
AtofStryker
commented
8 months ago Ah ha -- we're also seeing the error and using monaco! We're using it via
@monaco-react/loaderwhich loads the webworkers (and the majority of the code) from a CDN, so it doesn't get built via the webpack dev server by default. Not sure if monaco itself might be the issue, or if it's just super heavyweight and triggers a race in webpack.We don't use React nor "monaco" (we use Angular) and we have the same issue with component tests with the ChunkLoadError.. 🤔
This is definitely a webpack issue and should be framework agnostic.
AtofStryker
commented
8 months ago We have had good results in turning off splitChunks.chunks = 'all' and leveraging splitChunks.chunks = 'initial' to avoid async chunks. However, this isn't easy to do currently with @cypress/webpack-dev-server since the optimization option is overwritten . I created a feature request https://github.com/cypress-io/cypress/issues/29073 to allow users to configure this option. There are more details on a possible work around for this issue on the feature request and interested to see who gives it a try!
anilanar
commented
7 months ago We noticed that we get ChunkLoadErrors whenever CI machines run out of memory.
We don't know what's causing cypress to leak memory and whether if that's on our side or not. We use msw, therefore there's a chance that the service worker is leaking memory. Nevertheless, memory usage keeps increasing with no boundary and eventually hits 100% and crashes regardless of amount of memory, unless tests finish before being able to hit 100%.
anilanar
commented
7 months ago We noticed that service workers are leaking memory unless cypress is ran with experimentalSingleTabRunMode: true config.
anilanar
commented
7 months ago @AtofStryker using split chunks: 'initial' causes webpack node process to crash due to OOM. I'm guessing that happens with a large code base + uglifier.
AtofStryker
commented
7 months ago @AtofStryker using split chunks: 'initial' causes webpack node process to crash due to OOM. I'm guessing that happens with a large code base + uglifier.
It did consume more memory for us but it didn't cause the process to crash. It can see this being the case with a lot of CT tests. Out of curiosity does this happen locally or in CI and are you able to increase memory in any scenario?
anilanar
commented
7 months ago @AtofStryker Both.
I tried using NODE_OPTIONS="--max-old-space-size=32000" hoping that might make it go through it on my PC with 64gb RAM. I left for dinner, when I came back, it was using all 64gb RAM + swap and had to force restart the PC. So yeah, it seems it requires a lot of RAM. Looking at the default webpack config at @cypress/webpack-dev-server, it's not caused by uglifier either because it runs in development mode.
However, when I run cypress for a single spec, it only uses few hundred MBs of RAM.
I'll try splitting shared vendors/code, because I assume they are duplicated in all entries.
TomaszG
commented
7 months ago @AtofStryker, in my case (component tests for Next.js React components), setting splitChunks.chunks = 'initial' made things worse:
Loading chunk cypress-support-file failed. (error: http://localhost:8081/__cypress/src/cypress-support-file.js) on almost every run.Cypress: Config Manager hits 9-10 GB before the first test starts, compared to < 1GB with the default setting.anilanar
commented
7 months ago I think memory problems are caused by either source maps (webpackConfig.devtools) or webpack cache (webpackConfig.cache) or both.
I have a feeling chunk load errors happen due to swallowed OOM errors though. Did people try using high NODE_OPTIONS=--max-old-space-size? I noticed we started getting chunk load errors more often when I reduced CI machine size to a lower RAM class.
Nevertheless, I've given up on running a "dev server" on CI because that doesn't make sense to me as I previously expressed. Why run a long running build process and be prone to potential memory errors and harder to debug problems? Webpack dev server stores all generated assets in memory with a lot of overhead due to internal stats. Running webpack build and then serving static files from a simple static http server is much simpler, or it would've been simpler if cypress didn't pass spec name with a query parameter/header. Even then, serving them with an express server isn't too much trouble. Still, I wish cypress team took this path instead of potentially fighting an uphill battle.
So my implementation of a "custom dev server" still uses high memory during build but significantly reduced memory usage during test run, so I can even use different machine sizes for building and for running and save $$$.
Example custom dev server:
const { createCustomDevServer } = require('cypress-ct-custom-devserver');
exports.prebundleDevServer = createCustomDevServer(async ({ serveStatic }) => {
serveStatic('./build/cypress', '/__cypress/src');
return {
loadTest: async (spec, { loadBundle }) => {
const testPath = spec.relative.replace(/.(js|jsx|ts|tsx)$/, '.js');
loadBundle(testPath);
},
devServerPort: 8080,
onClose: () => {
compiler.close(err => {
console.error(err);
});
},
};
});This requires running webpack beforehand with a config like:
module.exports = {
entry: fs
.readdirSync('src', { recursive: true })
.filter(name => minimatch(name, '**/*.cy.{js,jsx,ts,tsx}'))
.map(name => `src/${name}`)
.reduce((acc, name) => {
acc[name.replace(/\.(js|jsx|ts|tsx)$/, '')] = ['./src/cypress/support.ts', `./${name}`];
return acc;
}, {}),
output: {
filename: '[name].js',
path: path.resolve('../build/cypress'),
publicPath: '/__cypress/src/',
},
// ...rest of webpack config
}Good luck everyone!
rockResolve
commented
6 months ago Getting ChunkLoadError often in our component test against a NextJs app.
From workaround tried cypress.config.ts
component: {
devServer: {
framework: 'next',
bundler: 'webpack',
webpackConfig: {
optimization: {
splitChunks: {
chunks: 'initial',
}, }, }, }Still failed with "Loading chunk packages_cypress_lib_index_ts failed."
anilanar
commented
6 months ago Just an update after 2 weeks: we don't have ChunkLoadErrors anymore with a custom "dev server" that serves prebundled static files.
mirobo
commented
6 months ago Any progress on the possible fix (by providing configuration options) according to #29073 ? We still encounter this issue quite often and a (possible) fix would be highly appreciated 🤗
ryanolson-aumni
commented
6 months ago This makes it difficult to depend on Cypress component tests for validation in CI. These chunk failures are making CI flakey. If we could get some official focus on this, that'd be appreciated. It's a real issue right now for us early adopters of Cypress component tests.
amir1218
commented
6 months ago We ended up reverting and pinning to 13.5.1 to avoid the crashes.
It's not a solid perm fix since we eventually will need to upgrade and pull in other critical bug fixes, but helps us keep our tests running.
ccannawright
commented
5 months ago We're seeing this on 13.2
werge2121
commented
5 months ago Any news on this, we are still pinned on v12 until this can be fixed?
chrisrhymes
commented
5 months ago I came across this issue today using Cypress 13.11.0. Not sure if this will help others, but for my particular test this was caused by me trying to set a env variable in the test to a dummy value in the beforeEach().
process.env.MY_CUSTOM_VAR = Cypress.env('MY_CUSTOM_VAR');The error went away when I commented out the variable in my .env.local file. Therefore, I updated the test to check if the variable is already set before trying to set it in the beforeEach().
if (!process.env.MY_CUSTOM_VAR) {
process.env.MY_CUSTOM_VAR = Cypress.env('MY_CUSTOM_VAR');
}This looks quite random to me with the process.env 🤔
In my case I never assign anything to process.env (I only read from process.env.LOCALAPPDATA when starting the browser with "headed"-mode, so it doesn't even apply for the tests running in CI). But in the tests and test setups I only ever use Cypress.env() to read configurations and never assign anything to process.env. I'm on 13.10.0 now..
Current behavior
After updating from
cypress@12.16.0tocypress@13.6.2, we are seeing this error occasionally on our CI runs.Unfortunately it is not reproducable locally, so I can not give you a repository to check. I know that this might make it hard to investigate but maybe somebody already fell over the same issue or others are experiencing it as well.
The test fails around 30% of the time.
Desired behavior
No response
Test code to reproduce
no reproduction repository
Cypress Version
13.6.2
Node version
18.19.0
Operating System
Ubuntu 22.04.3
Debug Logs
No response
Other
No response