search

cypress-io / cypress

Fast, easy and reliable testing for anything that runs in a browser.
https://cypress.io
MIT License
46.66k stars 3.16k forks source link

Memory leak in long running single test #4164

Open Konstruktour opened 5 years ago

Konstruktour commented 5 years ago

Current behavior:

Currently if a test runs the same actions repetitively over a long period of time, the memory of the cypress process increases until it crashes. (headless mode, video off, numTestsKeptInMemory 0) The sample to reproduce runs 15 minutes increasing the memory ~300MB. (the longer the run the more memory is used..)

Desired behavior:

Performance tests should be possible with cypress, without increasing memory and crashing.

Steps to reproduce: (app code and test code)

Config cypress.json

{
  "baseUrl": "http://localhost:8080/",
  "defaultCommandTimeout": 60000,
  "viewportWidth": 1280,
  "viewportHeight": 720,
  "numTestsKeptInMemory": 0,
  "video": false
}

Cypress test

describe('memory dummy test', () => {
    const endTime = Date.now() + 60000 * 15;

    before('visit', () => {
        cy.visit('/memory-test.html');
    });

    it('should test', () => {
        const test = () => {
            cy.get('#testinput').clear().type('hello world');
            cy.wait(2000);
            cy.then(() => {
                if (Date.now() < endTime) test();
            });
        };
        test();
    });
});

Simple static html site

<!DOCTYPE html>
<html lang="en">
<head>
</head>
<body>
    <input id="testinput">
</body>
</html>

Versions

Cypress 3.2.0 / 3.5.0 / 3.6.0 Windows 10

scharf commented 5 years ago

see https://github.com/cypress-io/cypress/issues/1906#issuecomment-502465837

brian-mann commented 5 years ago

I actually believe this issue will be fixed by this PR which will be merged in the next patch release: https://github.com/cypress-io/cypress/pull/4406

brian-mann commented 5 years ago

It was an oversight / bug that we were still taking snapshots when a single test even though numSnapshotsKeptInMemory was 0 - so it's expected that the memory would grow larger after each command runs - until the next test runs.

brian-mann commented 5 years ago

Closing this issue as duplicate because I'm very confident this is describing the same behavior as https://github.com/cypress-io/cypress/issues/4104

cypress-bot[bot] commented 5 years ago

Released in 3.3.2.

Konstruktour commented 4 years ago

@brian-mann / @jennifer-shehane: Retested it with version 3.6.0... still the same issue! :(

Konstruktour commented 4 years ago

@brian-mann / @jennifer-shehane: can this issue be reopened?

jennifer-shehane commented 4 years ago

This issue will be closed to further comment as the exact issue here was resolved and tested.

If you're experiencing a bug similar to this in Cypress, please open a new issue with a fully reproducible example that we can run. There may be a specific edge case with the issue that we need more detail to fix.

jennifer-shehane commented 4 years ago

I ran the provided code and was unable to reproduce the crashing described in the original issue. The spec ran for 15 minutes and finished to completion in Cypress 3.8.3 in Electron 78.

*Couldn't fit log in one screenshot

Screen Shot 2020-02-05 at 10 27 56 PM Screen Shot 2020-02-05 at 10 27 48 PM

There will be a new PR in 4.0 that will log process usage that would be helpful to see what is going on here memory wise https://github.com/cypress-io/cypress/pull/6171. How did you assess this is consuming more memory over time?

Konstruktour commented 4 years ago

thx for getting a look again at that issue... the crashing only occurs if the system gets out of memory.. the consumption increases over time, 15 min will be too less see the crash ... nevertheless you can track the memory consumption in the task manager.. eg the test started with ~100mb memory usage: image

after 45min it increased to ~360mb ... and rising... image

Konstruktour commented 4 years ago

@jennifer-shehane could u recreate the issue? any new findings?

Oliboy50 commented 4 years ago

@jennifer-shehane if you want another easily reproductible example => I guess I'm experiencing something similar in this project on cypress 4.10.0

it runs a single test which is pretty fast at the beginning and starts to be really slow after ~50 seconds of execution

I tried to use "numTestsKeptInMemory": 0 cypress configuration but it didn't help at all 🤷

if you want to try it out, please install node v12.x and run the following commands:

git clone https://github.com/Oliboy50/coinche.git
cd coinche

cd client
npm install
cd ..

cd server
npm install
cd ..

cd e2e
npm install
npm run dev

Cypress GUI will pop up and you'll be able to run the test I'm talking about

hope it will help (I'd love to have a faster test, one which does not make my computer burn ❤️)

jennifer-shehane commented 3 years ago

We're investigating some performance issues related to the rendering of the Command Log such as https://github.com/cypress-io/cypress/issues/6783 where I think the solution to that may also solve this issue. Will have to investigate once there is a proposed PR there.

pardamike commented 3 years ago

Our team has worked around this issue by simply using:

launchOptions.args.push('--max_old_space_size=1500');
launchOptions.args.push('--disable-dev-shm-usage');

inside of our plugins/index.js

Note that we are on an older version of Cypress and older image, and we are running this on Gitlab CI: package.json: "cypress": "^4.12.1" .gitlab-ci.yml (image): name: cypress/included:4.9.0

Hopefully this can help someone else 👍

Oliboy50 commented 3 years ago

@pardamike I've tried to put the following in my plugin.js file but it didn't help (at least not in chrome view):

module.exports = (on) => {
  on('before:browser:launch', (_, launchOptions) => {
    launchOptions.args.push('--max_old_space_size=1500');
    launchOptions.args.push('--disable-dev-shm-usage');
  });
};

🤷

pardamike commented 3 years ago

@Oliboy50 - Yea that looks good to me, for reference this is what we have in our plugins/index.js:

module.exports = (on, config) => {
    on('before:browser:launch', (browser = {}, launchOptions) => {
        if (browser.name === 'chrome') {
            launchOptions.args.push('--disable-web-security');
            launchOptions.args.push('--disable-site-isolation-trials');
            launchOptions.args.push('--max_old_space_size=1500');
            launchOptions.args.push('--disable-dev-shm-usage');
            return launchOptions;
        }
    });
}

You also may need to increase or decrease the max_old_space_size setting, 1500 worked for us but maybe you need more or less memory. I would try: 500, 1000, 2000, and 10000 and see if any of those work

If that does not work, I suggest trying out this logging package so you can see the exact error you are getting in the terminal when you run your pipeline: https://github.com/flotwig/cypress-log-to-output (for us, Cypress was swallowing the console errors)

You will need to use the "chromeWebSecurity": false, setting in your cypress/environments/cypress.{env}.json files, this logging package seems to make the browser ignore that flag for some reason

Note that we are on an older version of Cypress and older image, and we are running this on Gitlab CI: package.json: "cypress": "^4.12.1" .gitlab-ci.yml (image): name: cypress/included:4.9.0

kaysan13 commented 3 years ago

Hello,

After I made this modification, cypress displayed that the configuration is deprecated

Oliboy50 commented 3 years ago

Just updated cypress from 4.12.1 to 5.5.0 and it results in a huge performance improvement for my long running test 🎉 Thank you for your work 💪 => I think that memory leaks are gone in 5.5.0, so I guess this issue could be closed now (if @Konstruktour agrees)

4.12.1:

5.5.0:

norvinino commented 3 years ago

free memory from one test case to another in cypress

norvinino commented 3 years ago

@ brian-mann / @ jennifer-shehane Free memory from one test case to another in cypress

andrei22b commented 3 years ago

@Oliboy50 not really, i still have it in 6.0.0

andrei22b commented 3 years ago

For me the problem was that it was keeping zombie processes in memory on the docker container.

I fixed it with running dumb init on that container Note this is for who uses docker

ADD https://github.com/Yelp/dumb-init/releases/download/v1.2.2/dumb-init_1.2.2_x86_64 /usr/local/bin/dumb-init RUN chmod +x /usr/local/bin/dumb-init ENTRYPOINT ["dumb-init", "--"]

ronwongsk661 commented 3 years ago

Any update on memory leak

cuadllop commented 3 years ago

I am still having this issue too on 6.4.0

reuben-rodrigues commented 2 years ago

8.5.0 as well...

darkhillreturns commented 2 years ago

Still having the same issue in 9.6.1

darkhillreturns commented 2 years ago

Any updates on this from Cypress?

persianturtle commented 2 years ago

I'm also having a similar issue in 9.6.1.

JaLe29 commented 2 years ago

same problem 9.6.1

import { links } from './links'; // 500 items

context('Full version tests', () => {
    beforeEach(() => {
        cy.login();
    });

    links.forEach(link => {
        it(`should load page ${link}`, () => {
            cy.visit(link); 
            cy.wait(5000);
            cy.hasNotErrorMessage();
        });
    });
});

image

0618 commented 1 year ago

Still have this issue on 10.10.0

OmarEzQED commented 1 year ago

Also have the same issue on 10.10.0

BlueWinds commented 1 year ago

Doing a bit of consolidating of issues; I've closed several as duplicates of this in an effort to clean up all of our "the command log is slow in large tests" bugs.

This is something that's come up a bunch recently, and going to see if we can dedicate more time to looking into it soon.

mauroSoloLearn commented 1 year ago

Also have the same issue on 10.11.0 and chrome 107

dmolin commented 1 year ago

Any update on this? this is a huge road blocker in our project. We've long running tests that inevitably end up with the browser crashing bc of out of memory errors. Is there any actual activity/research currently done on this issue?

krisodb commented 1 year ago

Check https://github.com/cypress-io/cypress/discussions/25557

I'v also added env var CYPRESS_NO_COMMAND_LOG=1 on our pipeline. Not much added value having the command log for remote runs.

I've added some memory graphs on twitter: https://twitter.com/op_kris/status/1636675020180058112?s=46&t=DPClh6K4rCQElpP_CUCryg showing the difference between the usage of the command log or not.

CloneOfAlex commented 1 year ago

any updates ??

Error code: Out of Memory

nagash77 commented 1 year ago

@CloneOfAlex In order to give our engineers the best chance at recreating your problem, please create a reproducible example using a fork of Cypress Test Tiny. This gives us the best chance of seeing exactly what you are seeing and being able to investigate effectively.

StefanTegeltija commented 8 months ago

Could you please inform me if you haven't managed to reproduce this issue from 2019 so I can make reproducible example for you?

scharf commented 8 months ago

I have seen out or memory errors this recently

CloneOfAlex commented 8 months ago

@CloneOfAlex In order to give our engineers the best chance at recreating your problem, please create a reproducible example using a fork of Cypress Test Tiny. This gives us the best chance of seeing exactly what you are seeing and being able to investigate effectively.

Sadly I cant share our code

BUT -- on 16GB laptop when running with local runner , Cypress 13.3.3 / Node.js 20.10.0 - often runner crashes with out of memory error .. // hence forced to run only few scenarios at a time ..

Thanks!

StefanTegeltija commented 8 months ago

Hi @nagash77 , I have created reproducible example of this issue: https://github.com/StefanTegeltija/memory-leak-in-long-running-single-test, sorry I haven't used for of Cypres Test Tiny but it should not be hard to run my example. In order to run tests you just need to:

  1. Download this repo
  2. Run npm install
  3. Run npm run run:dev You will probably see issue in every run :) Please let me know is you need anything else :) Chrome version: Version 120.0.6099.71 (Official Build) (x86_64) Screenshot at Jan 05 10-11-18
LukasLewandowski commented 5 months ago

Any updates for memory problem?

CloneOfAlex commented 5 months ago

Hi @nagash77 , I have created reproducible example of this issue: https://github.com/StefanTegeltija/memory-leak-in-long-running-single-test, sorry I haven't used for of Cypres Test Tiny but it should not be hard to run my example. In order to run tests you just need to:

  1. Download this repo
  2. Run npm install
  3. Run npm run run:dev You will probably see issue in every run :) Please let me know is you need anything else :) Chrome version: Version 120.0.6099.71 (Official Build) (x86_64) Screenshot at Jan 05 10-11-18

What Node.js version you have installed ??

do node -v and let us know

Also -- you are using OLD Cypress version .. : (

image

Why dont you install:

Node.js 20.11.0 & Lates version of Cypress 13.7.0

StefanTegeltija commented 5 months ago

v16.18.1

CloneOfAlex commented 5 months ago

v16.18.1

You are using obsolete Node and Cypress versions ..

Install following and try again: Node.js 20.11.0 & Latest version of Cypress 13.7.0

CloneOfAlex commented 5 months ago

after installing latest Node and Cypress

also run

npm outdated

next mitigate outdated packages

run 

npm install
npm audit fix
npm outdated
StefanTegeltija commented 5 months ago

So you managed to fix this issue by these suggestions using my repo?

StefanTegeltija commented 5 months ago

BTW the newest version of cypress cannot be downloaded for some reason. npm WARN deprecated @babel/plugin-proposal-class-properties@7.18.6: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-class-properties instead.

npm WARN deprecated @babel/plugin-proposal-object-rest-spread@7.20.7: This proposal has been merged to the ECMAScript standard and thus this plugin is no longer maintained. Please use @babel/plugin-transform-object-rest-spread instead.
npm WARN deprecated intl-messageformat-parser@1.8.1: We've written a new parser that's 6x faster and is backwards compatible. Please use @formatjs/icu-messageformat-parser
npm WARN deprecated gherkin@5.1.0: This package is now published under @cucumber/gherkin
npm WARN deprecated cucumber-expressions@5.0.18: This package is now published under @cucumber/cucumber-expressions
npm WARN deprecated cucumber-expressions@6.6.2: This package is now published under @cucumber/cucumber-expressions
npm WARN deprecated cucumber@4.2.1: Cucumber is publishing new releases under @cucumber/cucumber
npm WARN deprecated puppeteer@9.1.1: < 21.5.0 is no longer supported
npm WARN deprecated core-js@2.6.12: core-js@<3.23.3 is no longer maintained and not recommended for usage due to the number of issues. Because of the V8 engine whims, feature detection in old core-js versions could cause a slowdown up to 100x even if nothing is polyfilled. Some versions have web compatibility issues. Please, upgrade your dependencies to the actual version of core-js.
npm ERR! code 1
npm ERR! path /Users/stefantegeltija/Desktop/projects/cs-qa-automation/node_modules/cypress
npm ERR! command failed
npm ERR! command sh -c node index.js --exec install
npm ERR! Installing Cypress (version: 13.7.0)
npm ERR! 
npm ERR! [STARTED] Task without title.
npm ERR! The Cypress App could not be downloaded.
npm ERR! 
npm ERR! Does your workplace require a proxy to be used to access the Internet? If so, you must configure the HTTP_PROXY environment variable before downloading Cypress. Read more: https://on.cypress.io/proxy-configuration
npm ERR! 
npm ERR! Otherwise, please check network connectivity and try again:
npm ERR! 
npm ERR! ----------
npm ERR! 
npm ERR! URL: https://download.cypress.io/desktop/13.7.0?platform=darwin&arch=x64
npm ERR! Error: Failed downloading the Cypress binary.
npm ERR! Response code: 502
npm ERR! Response message: Bad Gateway
npm ERR! 
npm ERR! ----------
npm ERR! 
npm ERR! Platform: darwin-x64 (22.4.0)
npm ERR! Cypress Version: 13.7.0
npm ERR! [FAILED] The Cypress App could not be downloaded.
npm ERR! [FAILED] 
npm ERR! [FAILED] Does your workplace require a proxy to be used to access the Internet? If so, you must configure the HTTP_PROXY environment variable before downloading Cypress. Read more: https://on.cypress.io/proxy-configuration
npm ERR! [FAILED] 
npm ERR! [FAILED] Otherwise, please check network connectivity and try again:
npm ERR! [FAILED] 
npm ERR! [FAILED] ----------
npm ERR! [FAILED] 
npm ERR! [FAILED] URL: https://download.cypress.io/desktop/13.7.0?platform=darwin&arch=x64
npm ERR! [FAILED] Error: Failed downloading the Cypress binary.
npm ERR! [FAILED] Response code: 502
npm ERR! [FAILED] Response message: Bad Gateway
npm ERR! [FAILED] 
npm ERR! [FAILED] ----------
npm ERR! [FAILED] 
npm ERR! [FAILED] Platform: darwin-x64 (22.4.0)
npm ERR! [FAILED] Cypress Version: 13.7.0

npm ERR! A complete log of this run can be found in: /Users/stefantegeltija/.npm/_logs/2024-03-19T14_38_18_486Z-debug-0.log
CloneOfAlex commented 5 months ago

OK - try other versions, example 13.3.3

StefanTegeltija commented 5 months ago

Man I tried all versions, and problem is still here, download my repo, try by yourself and I'm sure you will reproduce the issue immediately.