search

cypress-io / cypress

Fast, easy and reliable testing for anything that runs in a browser.
https://cypress.io
MIT License
47.02k stars 3.18k forks source link

Cypress slows down and then crashes during cypress run #3829

Closed ccorcos closed 5 years ago

ccorcos commented 5 years ago

Hey there, I'm new to Cypress and things we going well, but now things are starting to crash now that I have a larger test suite. I've noticed that this sort of issue was filed a bunch of times before (https://github.com/cypress-io/cypress/issues/431) but I'm just not sure what I should be doing to fix the issue...

Current behavior:

Here's an error report from Terminal when running with Electron 59. Things started to get pretty slow before it completely died.

<--- Last few GCs --->

[68129:0x7ff0b0854800]   198798 ms: Mark-sweep 2053.6 (2155.9) -> 2053.6 (2140.9) MB, 1574.1 / 78.1 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 1574 ms) last resort
[68129:0x7ff0b0854800]   200480 ms: Mark-sweep 2053.6 (2140.9) -> 2053.6 (2140.9) MB, 1681.6 / 78.6 ms  last resort

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0xa0bb29f87e9 <Window map = 0x2039cbd43f99>
    2: getInitialState [http://localhost:3000/app.js:415928] [bytecode=0xabd9ae04d21 offset=1](this=0x2e5ba689d021 <ResizeStore map = 0xcd2a0fbe271>)
    4: constructor(aka Store) [http://localhost:3000/app.js:418152] [bytecode=0xb271520bc51 offset=39](this=0x2e5ba689d021 <ResizeStore map = 0xcd2a0fbe271>)
    6: new constructor(aka ResizeStore) [http://localhost:3000/app.js:41...

Those constructors are objects in my code. I don't think there's a memory leak in my application code. I have a few custom commands that access some objects off of the window but I'm navigating around which should garbage collect everything. Not sure what Cypress is doing internally... Maybe its keeping everything around?

Also, happy to try whatever you want to see if we can get this working 👍 I really like everything else about Cypress!

Versions

ProductName: Mac OS X ProductVersion: 10.14 BuildVersion: 18A391

Cypress: 3.1.5

ccorcos commented 5 years ago

I split all of my tests into separate files. Each file has one describe and one it statement. When I run the tests together ("Run all specs"), it crashes. But when I run each one individually, they all work...

The more I think about it, I thing the memory leak is coming from my custom command... If the custom command holds the references to the objects between renders, then that would for sure cause a problem...

ccorcos commented 5 years ago

Here's an example command:

Cypress.Commands.add(
    "flushTransactions",
    { prevSubject: "optional" },
    subject => {
        cy.window()
            .then(async win => {
                const LocalTransactionQueue = win["LocalTransactionQueue"]
                await timeout(200)
                await LocalTransactionQueue.drain()
            })
            .then(() => subject)
    }
)

The LocalTransactionQueue holds a lot of object references and I'm not sure Cypress is letting them get GC'd...

ccorcos commented 5 years ago

I replaced all logic that accesses the window with waits... still runs out of memory and crashes...

Cypress.Commands.add(
    "flushTransactions",
    { prevSubject: "optional" },
    subject => {
        cy.wait(1000)
        .then(() => subject)
    }
)
ccorcos commented 5 years ago

I removed all custom commands. Still runs out of memory and crashes.

Everything I'm using Cypress vanilla command with Typescript now...

ccorcos commented 5 years ago

Its also worth mentioning that this is a problem on both Chrome and Electron

ccorcos commented 5 years ago

Hmm. I guess maybe this is expected?

https://github.com/cypress-io/cypress/issues/1077

ccorcos commented 5 years ago

After adding the "numTestsKeptInMemory": 1 option, running from the GUI succeeds without crashing. 🎉

ccorcos commented 5 years ago

Looks like Docker is still crashing in CircleCI though...

We detected that the Chromium Renderer process just crashed.

This is the equivalent to seeing the 'sad face' when Chrome dies.

This can happen for a number of different reasons:

- You wrote an endless loop and you must fix your own code
- There is a memory leak in Cypress (unlikely but possible)
- You are running Docker (there is an easy fix for this: see link below)
- You are running lots of tests on a memory intense application
- You are running in a memory starved VM environment
- There are problems with your GPU / GPU drivers
- There are browser bugs in Chromium

You can learn more including how to fix Docker here:

https://on.cypress.io/renderer-process-crashed
ccorcos commented 5 years ago 
df -h /dev/shm
Filesystem      Size  Used Avail Use% Mounted on
shm              30G  8.0K   30G   1% /dev/shm

Looks like there's plenty of shared memory...

Lakitna commented 5 years ago

I believe there could be issues when you run a lot of tests in GUI mode. How many tests are we talking here?

ccorcos commented 5 years ago

8 tests. Takes a total of 10 minutes to run.

ccorcos commented 5 years ago

Still crashes in headless mode though.

I split all 8 tests into separate files and things seems to be working fairly reliably at the moment...

DomDabrowski commented 5 years ago

Hello!

I have the same problem. I run 25 tests (12 files) with Chrome on my local machine. Whole execution takes about 12 minutes (estimated from running them all separately). Cypress starts to slow down circa about 8 minutes after start, then crashes.

Lakitna commented 5 years ago

Hmm now that I think about it, the other day I read about someone having issues with snapshots because of very large css files. #2366

Could this be related to your issues?

ccorcos commented 5 years ago

Hmm. No, but we have about 4MB of gzipped javascript...

DomDabrowski commented 5 years ago

Adding "numTestsKeptInMemory": 1 to my cypress.json helped so looks like Cypress is short of memory after a few executed tests.

tnrich commented 5 years ago

What is interesting to me is that I find that running cypress open locally slows down as I re-run the same single test. It seems like there must be a memory leak there..? Does anyone else experience a slow down when re-running a single test?

maksimkarpov commented 5 years ago

My test suite of about 50 tests crashes each other run when with Electron browser. Works fine with chrome. In logs it just says it crashed

jennifer-shehane commented 5 years ago

We've been trying to recreate this with no success. We believe this is happening and just want to track down why.

If you are having crashing issues during cypress open - please comment in this issue: https://github.com/cypress-io/cypress/issues/1937

Please can anyone provide the test code to run to reproduce the situation where running multiple tests slows down and eventually crashes while running in cypress run. 🙏

idanRiski commented 5 years ago

Happens for me as well by docker. I've didn't manage to solve this problem and I tried a lot of potential solutions. :/

Frituurpanda commented 5 years ago

I understand that this might be hard to replicate as these tests are client specific but I'm observing the exact same behaviour with my instance. Currently running a bunch of Jira UI tests on CircleCI and this simply grinds to a halt after a while. Same goes for cypress run or cypress open.

scharf commented 5 years ago

@jennifer-shehane see https://github.com/cypress-io/cypress/issues/1906#issuecomment-502465837 on how to reproduce the problem

marinakuranda commented 5 years ago

Seem to be having the same issue. Headless mode, Electron, Cypress 3.3.2, video false. Running one and the same test 200 times with

Cypress._.times(200, (i) => {
  it(`num ${i + 1} - test the thing conditionally`, () => {

The test runs well for the first 125 times and takes about 13 sec to complete. After that, the run time starts to slow down gradually with each test, more and more, reaching up to 3 min to complete. During the last 75 times, the test fails at random places due to not finding elements and timing out. Happens regardless of which test is being run.

Hope this sheds some light on reproducing the issue.

LeroyDoornebal commented 5 years ago

I'm having the exact same issues ad described above and I am running Cypress in Scrutinizer. Last test always grinds to a halt (never passes) with the following output.

The steps files are Typescript and I'm using the cypress-cucumber-preprocessor. My error is a little bit different, but boils down to the same: 

<--- Last few GCs --->

[4486:0x333a1c083000]   509707 ms: Mark-sweep 2063.5 (2189.4) -> 2063.5 (2182.9) MB, 2143.1 / 0.0 ms  (+ 0.0 ms in 0 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 2143 ms) last resort GC in old space requested
[4486:0x333a1c083000]   511967 ms: Mark-sweep 2063.5 (2182.9) -> 2063.5 (2182.9) MB, 2258.9 / 0.0 ms  last resort GC in old space requested

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x33513baab0a1 <JSObject>
    2: /* anonymous */(aka /* anonymous */) [/home/scrutinizer/build/node_modules/typescript/lib/typescript.js:29145] [bytecode=0x3f9619d73699 offset=3](this=0x3d2d68982311 <undefined>,ElementKind=0x3fa2de58d7c1 <Object map = 0x17fa694023b9>)
    4: bindObjectLiteralExpression(aka bindObjectLiteralExpression) [/home/scrutinizer/build/node_modules/typescript/lib/typescript.js:29147] [bytecode=0x3...

Next to that I get the following stack trace:

The following error was thrown by a plugin. We've stopped running your tests because a plugin crashed.

Error [ERR_IPC_CHANNEL_CLOSED]: channel closed
    at ChildProcess.target.send (internal/child_process.js:588:16)
    at Object.send (/home/scrutinizer/.cache/Cypress/3.4.1/Cypress/resources/app/packages/server/lib/plugins/util.js:34:27)
    at EventEmitter.<anonymous> (/home/scrutinizer/.cache/Cypress/3.4.1/Cypress/resources/app/packages/server/lib/plugins/preprocessor.js:60:18)
    at emitOne (events.js:121:20)
    at EventEmitter.emit (events.js:211:7)
    at Object.removeFile (/home/scrutinizer/.cache/Cypress/3.4.1/Cypress/resources/app/packages/server/lib/plugins/preprocessor.js:108:19)
    at Object.options.onBrowserClose (/home/scrutinizer/.cache/Cypress/3.4.1/Cypress/resources/app/packages/server/lib/open_project.js:100:30)
    at EventEmitter.<anonymous> (/home/scrutinizer/.cache/Cypress/3.4.1/Cypress/resources/app/packages/server/lib/browsers/index.js:131:21)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:111:20)
    at EventEmitter.emit (events.js:208:7)
    at BrowserWindow.<anonymous> (/home/scrutinizer/.cache/Cypress/3.4.1/Cypress/resources/app/packages/server/lib/browsers/electron.js:248:29)
    at Object.onceWrapper (events.js:315:30)
    at emitOne (events.js:121:20)
    at BrowserWindow.emit (events.js:211:7)

As it is kind of hard to make this error reproducible (due to private repo). What I can do is share the test that is failing, but as you'll see there's nothing exciting there...

feature file

Feature: Page not found

    I want to be notified when a page does not exist

    Scenario: Display the page not found page
        Given I navigate to a page that does not exist
        Then I should see a notification that the page is not found

step definition

import { Given, Then } from 'cypress-cucumber-preprocessor/steps';
import { testHooks } from 'src/testing/testHook';

Given('I navigate to a page that does not exist', () =>
  cy.visit('http://localhost:3000/a-non-existant-page')
);

Then('I should see a notification that the page is not found', () => {
  cy.getById(testHooks.NotificationPage).contains('Page not found');
});

Where the getById command is simply a wrapper around get

Cypress.Commands.add('getById', testHookId =>
  cy.get(`[data-test-id="${testHookId}"]`, { timeout: 20000 })
);
ccorcos commented 5 years ago

It has been 5 months with no progress on this issue. I would not rely on Cypress fixing this issues. I've moved my company on to using the selenium-webdriver package. The only (small) hurdles were implementing my own retry-timeout and wrapping errors into better error messages.

For now, I'm going to close this issue because it is not making progress and is giving people a false sense of hope.

srinu-alef commented 5 years ago

It has been 5 months with no progress on this issue. I would not rely on Cypress fixing this issues. I've moved my company on to using the selenium-webdriver package. The only (small) hurdles were implementing my own retry-timeout and wrapping errors into better error messages.

For now, I'm going to close this issue because it is not making progress and is giving people a false sense of hope.

I am also of the same opinion and finding hard to go back to other tool after investing lot of time and still there is no light in this darkness.... There is no option and just to move away from this. I also their efforts but we cannot reply on this type of feature or support. It has been 3 years this bug has been reported and still people are struggling to overcome with this problem.

ccorcos commented 5 years ago

@srinu-alef and anyone else struggling with this, I've been using Selenium without any issues for months now. I created a monadic API similar to Cypress that is easy to use and easy to migrate over. Please feel free to use it and spread the word:

https://gist.github.com/ccorcos/5372e1f946927d5043f070fb9260fcea

craig-dae commented 4 years ago

I'm having constant issues with Cypress freezing on my AWS Workspace. It just started happening a couple weeks ago. It cases tests to fail due to timeout.

Is this being worked on?

ccorcos commented 4 years ago

They've constantly avoided addressing this issue. The gist I mentioned here is almost a drop-in replacement that uses Selenium instead.

jennifer-shehane commented 4 years ago

@ccorcos We have not been avoiding this issue. We cannot begin work on an issue where we have not been given a reliable way to reproduce - which no one has provided. We have thousands of tests and also performance tests, so when these issues arise it is very likely an edge case involving something specific to your tests, your application under test or environment. We need to be able to replicate this completely in order to address it.

@craig-dae @srinu-alef @Frituurpanda @idanRiski For anyone else experiencing a 'slow down/crash', open a new issue with a completely reproducible example since the original author has closed this issue.

We really want to fix performance issues. They are high priority for us.

@marinakuranda I am not able to detect any slowdown in Cypress in 3.8.1 with your example. Video below. Please provide more detail if necessary (for example, does this only happen in cypress run)

@LeroyDoornebal Thanks for the detail, unfortunately this example cannot be run on our machines so cannot be verified by us.

@scharf You mentioned in your comment that the link is how to 'reproduce the problem', but you provided zero test code for us to run to reproduce the problem. Can you please open a new issue with the test code required to run to see the performance issues you've described.

craig-dae commented 4 years ago

@jennifer-shehane I can't reproduce reliably. It started happening sometime in the past few weeks. We wrote about half a dozen integration tests on a new project before it started happening.

I'll run a test manually, using the UI. It's a coin-flip whether it freezes or runs smoothly. It will usually freeze when the app is loading initially. It will freeze less often, after the preloading animation, somewhere usually near the beginning of the tests. It will usually complete the test if gets through maybe the first 25% of the test.

It will sometimes freeze for 10s and then unfreeze. This causes the test to fail due to timeout. It usually freezes long enough for me to grow impatient and kill it. I'll then go back to the cypress UI, select the test, and run it again, and it's another coin-flip as to whether it gets through.

This is not me doing hundreds of tests. This is freezing when I run one single test. The test clicks a few buttons and fills out a form. The most complicated thing in it is cy.server() and some cy.wait(@example) calls to wait for some web services to return before continuing.

I have not seen the test fail when I do npx cypress run, only when I do npx cypress open.

I'm testing an Angular project, if you think that might be relevant.

Please advise on what else I can do to help you with this. I can open a new ticket if that helps.

craig-dae commented 4 years ago

@jennifer-shehane I'm going to open a new ticket with what I wrote above. If I could reproduce the problem reliably, I would. I'll need you to help me go through some troubleshooting steps.

jennifer-shehane commented 4 years ago

@craig-dae Ok, also make sure to check out and isolate whether this is an existing performance issue that is a bug in Chrome 79 here: https://github.com/cypress-io/cypress/issues/6023#issuecomment-574011973

craig-dae commented 4 years ago

@jennifer-shehane I think that the issue on this page is different from the other thread that I started. I don't have the problem with cypress run. I do have the problem with cypress test.

I am confident that the issue is with Chrome 79, and not with Cypress, for my issue. Thanks!

nzn-idt commented 4 years ago

First of all a big thanks to Brian, Chris, Gleb, Jennifer and all the contributors for this fantastic and so-easy-to-use framework. Well done!

I've also been plagued by the slowdown from the start since I have to run dozens of test from the GUI during dev-time so I can see what's going on in real time, but I've discovered that the slowdown can be abolished - no matter how big and how numerous are the tests - when the "test" collapsible area under the test name is closed. image

Since it's a pain to close each one as we get to each test, I've found a quick hack you can do in the code to keep them closed. In the file Cypress\resources\app\packages\runner\dist\cypress_runner.js a look for "var Hook = Object", a little below that (line 102918 in version 3.8.3, line 156012 in version 4.5.0) change the isOpen field from "true" to "false": image Now all the tests run quickly and smoothly without any slowdown.

tnrich commented 4 years ago

@nzn-idt where can I find that file Cypress\resources\app\packages\runner\dist\cypress_runner.js ?

nzn-idt commented 4 years ago

Assuming a Windows installation Cypress\ is the directory containing your Cypress.exe file (i.e. the topmost level).

dudziakm commented 4 years ago

for me, a good solution was to turn off video recording and compressing ("video": false, in cypress.json). Of course, this is not a perfect solution :(

cjoshmartin commented 4 years ago

Cypress seems to slow down/freeze, when I open the chrome(V84) dev tools.

natural411 commented 4 years ago

I just updated to 5.4.0 and mine are working again (but i also closed dev tools). not sure for how long though, definitely a memory leak somewhere

moontemplar212 commented 2 years ago

First of all a big thanks to Brian, Chris, Gleb, Jennifer and all the contributors for this fantastic and so-easy-to-use framework. Well done!

I've also been plagued by the slowdown from the start since I have to run dozens of test from the GUI during dev-time so I can see what's going on in real time, but I've discovered that the slowdown can be abolished - no matter how big and how numerous are the tests - when the "test" collapsible area under the test name is closed. image

Since it's a pain to close each one as we get to each test, I've found a quick hack you can do in the code to keep them closed. In the file Cypress\resources\app\packages\runner\dist\cypress_runner.js a look for "var Hook = Object", a little below that (line 102918 in version 3.8.3, line 156012 in version 4.5.0) change the isOpen field from "true" to "false": image Now all the tests run quickly and smoothly without any slowdown.

Cypress 9.4.1 line 189278 ( C:\Users\\AppData\Local\Cypress\Cache\9.4.1\Cypress\resources\app\packages\runner\dist\cypress_runner.js ) => isOpen: false where headerClass: "hook-header",