search

cypress-io / cypress

Fast, easy and reliable testing for anything that runs in a browser.
https://cypress.io
MIT License
46.87k stars 3.17k forks source link

Tests causing CI agents to run out of memory on cypress 8.2 & above #17853

Closed vaibhavganatra closed 2 years ago

vaibhavganatra commented 3 years ago

Current behavior

The Azure CI Agent machine running out of memory when running cypress tests on 8.2 & above on Azure Pipeline CI agents (Linux OS). The cypress dashboard eventually times out as well.

<--- Last few GCs --->

[31:0x1c7300000000]  8599171 ms: Mark-sweep 1835.4 (2072.0) -> 1832.6 (2071.6) MB, 205.8 / 0.0 ms  (average mu = 0.263, current mu = 0.223) allocation failure scavenge might not succeed
[31:0x1c7300000000]  8599492 ms: Mark-sweep 1835.0 (2072.0) -> 1832.6 (2071.6) MB, 294.6 / 0.0 ms  (average mu = 0.166, current mu = 0.082) allocation failure scavenge might not succeed

<--- JS stacktrace --->

The Test Runner unexpectedly exited via a exit event with signal SIGSEGV

Please search Cypress documentation for possible solutions:

https://on.cypress.io

Check if there is a GitHub issue describing this crash:

https://github.com/cypress-io/cypress/issues

Consider opening a new issue.

----------

Platform: linux (Debian - 10.10)
Cypress Version: 8.3.0

Desired behavior

No response

Test code to reproduce

This has been consistently reproducible on 8.2 and 8.3. Running the same spec files on the same agents on version 8.1 and below works as expected.

Cypress Version

8.3

Other

No response

cellog commented 3 years ago

a note: in a much earlier version of cypress, we fixed this by adding

  "numTestsKeptInMemory": 1,

to our cypress-ci.json, this may help if you're not already doing this

vaibhavganatra commented 3 years ago

I tried that and it is still failing with the same error.

stychu commented 3 years ago

I can observe similar issues since 8.2 and higher. My AZURE pipelines just run until the pipeline timeout occurs once I upgrade above 8.1.

On Cypress 8.1 everything works flawlessly

jennifer-shehane commented 3 years ago

Is everyone in this thread experiencing this issue in Azure only?

jennifer-shehane commented 3 years ago

Could anyone run Cypress in debug mode mode and print the entire set of logs here?

cupcakepanda89 commented 3 years ago

I experience this when running cypress in aws codebuild too

VadimSkuratovskyi commented 3 years ago

@jennifer-shehane I've got the same issue with 8.3.1 on Azure Linux agents: 

2021-09-10T08:46:44.512Z cypress:cli checking environment variables
2021-09-10T08:46:44.513Z cypress:cli checking if executable exists /home/vsts/.cache/Cypress/8.3.1/Cypress/Cypress
2021-09-10T08:46:44.514Z cypress:cli Binary is executable? : true
2021-09-10T08:46:44.515Z cypress:cli binaryDir is  /home/vsts/.cache/Cypress/8.3.1/Cypress
2021-09-10T08:46:44.515Z cypress:cli Reading binary package.json from: /home/vsts/.cache/Cypress/8.3.1/Cypress/resources/app/package.json
2021-09-10T08:46:44.517Z cypress:cli Found binary version 8.3.1 installed in: /home/vsts/.cache/Cypress/8.3.1/Cypress
2021-09-10T08:46:44.519Z cypress:cli could not read binary_state.json file at "/home/vsts/.cache/Cypress/8.3.1/binary_state.json"
2021-09-10T08:46:44.520Z cypress:cli {}
2021-09-10T08:46:44.520Z cypress:cli is Verified ? undefined
2021-09-10T08:46:44.520Z cypress:cli running binary verification check 8.3.1
It looks like this is your first time using Cypress: 8.3.1

 cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'plugin'      │      2       │      '3757, 14814'       │   43.01    │     47.46      │  2574.7  │   1243.49    │   2654.23   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '3354'          │     5      │      4.31      │  221.75  │    249.44    │    265.1    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │ '3356, 3515, 3357, 3676' │   102.5    │     25.85      │  193.93  │    201.46    │   204.38    │
  cypress:server:util:process_profiler │    3    │    'Electron'     │      1       │         '14802'          │    0.05    │     14.75      │  118.04  │    242.54    │   712.32    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │      '14832, 14833'      │     0      │       0        │   3.48   │     3.45     │    3.56     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      10      │           '-'            │   150.56   │     91.66      │ 3111.91  │    1919.5    │   3262.54   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +11s
  cypress:server:util:process_profiler current & mean memory and CPU usage by process group:
  cypress:server:util:process_profiler ┌─────────┬───────────────────┬──────────────┬──────────────────────────┬────────────┬────────────────┬──────────┬──────────────┬─────────────┐
  cypress:server:util:process_profiler │ (index) │       group       │ processCount │           pids           │ cpuPercent │ meanCpuPercent │ memRssMb │ meanMemRssMb │ maxMemRssMb │
  cypress:server:util:process_profiler ├─────────┼───────────────────┼──────────────┼──────────────────────────┼────────────┼────────────────┼──────────┼──────────────┼─────────────┤
  cypress:server:util:process_profiler │    0    │     'plugin'      │      2       │      '3757, 14814'       │   92.19    │      48.1      │ 2772.17  │   1265.33    │   2772.17   │
  cypress:server:util:process_profiler │    1    │     'cypress'     │      1       │          '3354'          │    0.95    │      4.26      │  222.39  │    249.05    │    265.1    │
  cypress:server:util:process_profiler │    2    │ 'electron-shared' │      4       │ '3356, 3515, 3357, 3676' │    0.48    │     25.49      │  193.93  │    201.36    │   204.38    │
  cypress:server:util:process_profiler │    3    │    'Electron'     │      1       │         '14802'          │    0.67    │     14.55      │  115.13  │    240.66    │   712.32    │
  cypress:server:util:process_profiler │    4    │      'other'      │      2       │      '14987, 14988'      │     0      │       0        │   3.41   │     3.45     │    3.56     │
  cypress:server:util:process_profiler │    5    │      'TOTAL'      │      10      │           '-'            │   94.29    │      91.7      │ 3307.03  │   1939.05    │   3307.03   │
  cypress:server:util:process_profiler └─────────┴───────────────────┴──────────────┴──────────────────────────┴────────────┴────────────────┴──────────┴──────────────┴─────────────┘ +10s

<--- Last few GCs --->

[3757:0x3f6200000000]   741154 ms: Scavenge 1998.0 (2061.2) -> 1996.5 (2063.2) MB, 14.8 / 0.0 ms  (average mu = 0.797, current mu = 0.427) allocation failure 
[3757:0x3f6200000000]   741216 ms: Scavenge 2000.4 (2063.2) -> 1999.0 (2067.2) MB, 29.4 / 0.0 ms  (average mu = 0.797, current mu = 0.427) allocation failure 
[3757:0x3f6200000000]   742915 ms: Mark-sweep 2004.3 (2067.5) -> 2001.3 (2074.0) MB, 1670.7 / 0.0 ms  (average mu = 0.650, current mu = 0.179) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
ikrasnikov commented 3 years ago

I have an Agular project and using Cypress for e2e testing. I also have an issue with memory.

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory 1: 0x108316625 node::Buffer::New(v8::Isolate*, char*, unsigned long, void (*)(char*, void*), void*) [/Users/maksim/Library/Caches/Cypress/8.3.1/Cypress.app/Contents/Frameworks/Electron Framework.framework/Versions/A/Electron Framework]

To reproduce the problem I made test project In my case, the problem goes when the 390+ test file is loading.

Platform: Ubuntu 20.4 NodeJs: 16.9.1 npm: 7.21.1 Cypress: 8.4.1 cypress-cucumber-preprocessor: 4.2.0

stychu commented 3 years ago

Any updates? I wish I could upgrade from 8.1 as I can see significant performance boost but this issue forces me to be locked to 8.1

ikrasnikov commented 3 years ago

@stychu there is no such issue in 8.1?

simongarciam commented 3 years ago

Tried to upgrade from cypress 7.2.0 to 8.6.0 and also found that the tests started consuming more resources from version 8.2.0 onwards (I pretty much tried all versions in between). Ended up pinning version to 8.1.0 but really want to go to at least 8.3.0 to upgrade chrome from version 88.

Testing React App + cypress 8.5.0 + chrome 94 in a container in pipeline using AWS using m5.xlarge instance, (16 GB ram). switching to a m5.2xlarge (32 GB) makes test pass but this just hides the problem and locally the spec with quite a bit of tests just crashes chrome.

Running headless, there's no log saying chrome crashed, cypress just hangs. Running in headed mode, I can see he browser crash and cypress just hangs.

stychu commented 3 years ago

@stychu there is no such issue in 8.1?

@ikrasnikov Yes I run 8.1 without the issues. As long as I try to bump it up to 8.2 and above all tests hang out and timeout azure pipeline.

ikrasnikov commented 3 years ago

@stychu thank you! But in my case did not help to decrease the version to 8.1. Still have the same issue after 390 test file.

stychu commented 2 years ago

Any updates? Is it working ?

BlueWinds commented 2 years ago

Vadim's logs suggest that it's the plugins child process which is consuming memory, which is very helpful in at least giving us a place to start looking.

It's very strange that this begins occurring between 8.1 and 8.2. If 8.2 ran but 8.3 had issues, there'd be a variety of possible culprits (starting with the bundled electron and node versions), but 8,2 doesn't have anything that immediately jumps out at me as a potential cause.

9.0.0 does contain an electron and bundled node version bump, it might be worth trying. It also now uses nodeVersion: 'system' as the default rather than bundled as it used to.

ikrasnikov commented 2 years ago

@BlueWinds, tried with version 9.0.0. No changes. Run fails on 295 test file. Here is the test project where the problem is reproduced

BlueWinds commented 2 years ago

@BlueWinds, tried with version 9.0.0. No changes. Run fails on 295 test file. Here is the test project where the problem is reproduced

Thank you for the reproducible example! Having an easy way for us to reproduce issues is always the first step in figuring things out. :+1:

@chrisbreiding - This is an ongoing issue I looked at last time and had to pass on because we didn't have good repro steps.

ikrasnikov commented 2 years ago

@BlueWinds but have already posted this test project before, in my comment Sep 24 :)

BlueWinds commented 2 years ago

@BlueWinds but have already posted this test project before, in my comment Sep 24 :)

Yeah, I see it now. Lots of issues to keep up with and not enough time to give them all the attention they deserve.

LoaderB0T commented 2 years ago

I have the same issue with a self-hosted Azure DevOps Agent. Fails after about 300 tests. Downgrading to 8.1.0 resolved the issue.

Interestingly enough, this issue was not always there with this version. It appeared out of nowhere months after updating to 8.4 but was persistent ever since.

cupcakepanda89 commented 2 years ago

Do we have any update on this will be fixed soon 😢

emilyrohrbough commented 2 years ago

Appears to be related to: https://github.com/cypress-io/cypress/issues/15508

vaibhavganatra commented 2 years ago

We have been stuck on 8.1 due to this issue. Can someone please prioritze this issue?

ikrasnikov commented 2 years ago

Tried to update to actual versions

"cypress": "9.3.1",
"cypress-cucumber-preprocessor": "4.3.1",
"typescript": "4.5.5"

Pushed changes to the test project. Still have the same issue

jeffchew commented 2 years ago

This just started happening in my project, currently only in a particular PR so trying to troubleshoot it and came across this issue. One of the test runs (in Github Actions):

https://github.com/carbon-design-system/carbon-for-ibm-dotcom/runs/4970182640?check_suite_focus=true

This is the PR where this is happening (https://github.com/carbon-design-system/carbon-for-ibm-dotcom/pull/8147), if it helps to see if a particular change is causing this. Our project was running fine on cypress v9.2.1 before this PR. Upgrading to latest version (9.3.1) didn't seem to help anything.

testerswieton commented 2 years ago

If you want a solution for: "[3078:0127/180408.450621:ERROR:node_bindings.cc(143)] Fatal error in V8: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory"

  1. Update the node version in Dockerfile eg: FROM node:17.3-slim

  2. Add memory for NODE in the docker-compose file used by GHA: services: cypress: environment:

    • NODE_OPTIONS="--max-old-space-size=8192"
jeffchew commented 2 years ago

If you want a solution for: "[3078:0127/180408.450621:ERROR:node_bindings.cc(143)] Fatal error in V8: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory"

  1. Update the node version in Dockerfile eg: FROM node:17.3-slim

  2. Add memory for NODE in the docker-compose file used by GHA: services: cypress: environment:

    • NODE_OPTIONS="--max-old-space-size=8192"

Thanks, I'm not using docker though, just running cypress tests on a node project in Github Actions. I can't upgrade node since our project has some dependency limitations at the moment.

I tried NODE_OPTIONS but that didn't work.

jeffchew commented 2 years ago

Just an update, I rolled my project back to Cypress 8.1 for now until this can be resolved. All tests run fine on that version.

handerss-spotfire commented 2 years ago

Also stuck on 8.1 due to this issue. Using node 17.3 and setting NODE_OPTIONS="--max-old-space-size=8192" before launching cypress does not fix the issue for us.

cupcakepanda89 commented 2 years ago

I updated to the latest cypress 9.4.1 and also use a latest version of cypress docker image public.ecr.aws/cypress-io/cypress/browsers:node14.17.0-chrome88-ff89 seems fix the issue on AWS CI https://gallery.ecr.aws/cypress-io/cypress/browsers

handerss-spotfire commented 2 years ago

Using latest cypress but basing the image off of public.ecr.aws/cypress-io/cypress/browsers:node14.17.0-chrome88-ff89 does not fix the issue either.

LoaderB0T commented 2 years ago

So we recently updated all our self-hosted build agents to completely new hardware with way more resources and still got the error. So to me, this indicates that this is not a hardware constraint (like an actual out of memory due to hardware limitations) but rather some software quirk. This is the actual error message I get:

<--- Last few GCs --->

[10856:0000780A0016C000]  2303795 ms: Scavenge 3420.3 (3538.7) -> 3420.3 (3539.7) MB, 4.1 / 0.0 ms  (average mu = 0.148, current mu = 0.000) allocation failure 
[10856:0000780A0016C000]  2303797 ms: Scavenge 3420.3 (3539.7) -> 3420.3 (3539.7) MB, 2.0 / 0.0 ms  (average mu = 0.148, current mu = 0.000) allocation failure 

<--- JS stacktrace --->

[10856:0328/173405.737:ERROR:node_bindings.cc(143)] Fatal error in V8: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Works just fine with v 8.1.0

stychu commented 2 years ago

@jennifer-shehane How this is still the issue when we hit almost 10 version of current cypress (almost 2 major versions and this still not solved/looked)

handerss-spotfire commented 2 years ago

We "solved" this by increasing the RAM on the machine. In 8.1 we could get away with very little memory, but now we need 16GB+.

If this a blocker for someone then it should be pretty easy to aid in finding the bug by doing a git-bisect on the changes from 8.1 to 8.2 (there aren't that many). I didn't get around to it since bumping the memory solved this for our relatively small test suite.

vaibhavganatra commented 2 years ago

@handerss-tibco Please note that we run our tests on default azure pipeline agents which come with default RAM and cannot be configured to increase the RAM. @jennifer-shehane This issue has stopped us from upgrading to latest cypress versions and have been stuck on 8.1 since August 2021. Can you please share any updates with us on this issue and if this can be prioritized?

jeffchew commented 2 years ago

Our team also runs on linux Github Actions runners which give only 7GB of RAM. We are also stuck on 8.1.

1r00t commented 2 years ago

I have enough memory (32GB). My cypress runs in WSL2 and crashes as soon as maxMemRssMb reaches ~3500MB. I tried various ways to give it more memory but nothing seems to work. I tried --max_old_space_size=16384 on chrome. I tried increasing memory in .wslconfig and much more. Nothing helps

DJSdev commented 2 years ago

Out of curiosity, I saw the note on the v10.0.0 release notes that a memory leak was fixed for e2e tests run using cypress run. Can anyone confirm if updating to v10+ resolved this issue? Someone at cypress maybe? @brian-mann

Related pull request: https://github.com/cypress-io/cypress/pull/19915

Release notes: https://docs.cypress.io/guides/references/changelog#10-0-0

L4B0MB4 commented 2 years ago

I got a out of memory exception in Chrome running on Cypress v10.1.0. Already have the "keep one test in memory" option enabled. Switched to edge and ran a few test files. Started with around 1GB memory usage and am now on steady 4GB (only running cypress and nothing more on edge). Its definitely seems like a memory leak as it just does not free the memory even after finishing a test run

stychu commented 2 years ago

Doesnt seem like the issue is fixed. I will check in azure in a moment but locally after migrating to cypress 10 when running the test my MAC is skyrocketing with CPU usage and it spins off the fans like crazy where as on 8.1 its quiet like a baby. 280% CPU usage for cypress process

intellix commented 2 years ago

Could this be caused by this? https://github.com/nodejs/node/issues/35889#issuecomment-1129293091 - which has troubled Jest users for a long time as well

stychu commented 2 years ago

As I though and process just hangs. This happens on azure alto there are some console errors which were not present in versions between 8.2 and 10


[30:0614/123746.553614:ERROR:zygote_host_impl_linux.cc(263)] Failed to adjust OOM score of renderer with pid 201: Permission denied (13)
[201:0614/123746.588096:ERROR:sandbox_linux.cc(377)] InitializeSandbox() called with multiple threads in process gpu-process.
[201:0614/123746.592132:ERROR:gpu_memory_buffer_support_x11.cc(44)] dri3 extension not supported.
Couldn't find tsconfig.json. tsconfig-paths will be skipped
Couldn't determine Mocha version

================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:        10.1.0                                                                         │
  │ Browser:        Electron 100 (headless)                                                        │
  │ Node Version:   v14.17.0 (/usr/local/bin/node)                                                 │
  │ Specs:          12 found (...)                                                                            │
  │ Searched:       cypress/tests/integration/**/*.ts                                              │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘

-----> Cleaning cypress/reports <-----

────────────────────────────────────────────────────────────────────────────────────────────────────

  Running:  ....                                                                (1 of 12)
[30:0614/123750.070030:ERROR:zygote_host_impl_linux.cc(263)] Failed to adjust OOM score of renderer with pid 406: Permission denied (13)
[30:0614/123750.287987:ERROR:cert_verify_proc_builtin.cc(681)] CertVerifyProcBuiltin for redirector.gvt1.com failed:
----- Certificate i=0 (OU=Cypress Proxy Server Certificate,O=Cypress Proxy CA,L=Internet,ST=Internet,C=Internet,CN=redirector.gvt1.com) -----
ERROR: No matching issuer found

[30:0614/123750.419943:ERROR:cert_verify_proc_builtin.cc(681)] CertVerifyProcBuiltin for app failed:
----- Certificate i=0 (OU=Cypress Proxy Server Certificate,O=Cypress Proxy CA,L=Internet,ST=Internet,C=Internet,CN=app) -----
ERROR: No matching issuer found

[30:0614/123751.069481:ERROR:cert_verify_proc_builtin.cc(681)] CertVerifyProcBuiltin for ... failed:
----- Certificate i=0 (OU=Cypress Proxy Server Certificate,O=Cypress Proxy CA,L=Internet,ST=Internet,C=Internet,CN=r...) -----
ERROR: No matching issuer found
stychu commented 2 years ago

Come-on we need this to be fixed ASAP there are multiple features from Cypress v10 that would simplify so much things and I want to use them but Im unable to due this bug.... @cellog @jennifer-shehane

DJSdev commented 2 years ago

Hey, my company has this issue too and I think I found the cause and am close to having a PR ready to fix this.

Setup

In my environment, I found this out of memory issue only happens in Chrome, it doesn't seem to occur in FireFox. Hopefully someone might be able to confirm this for me, as this would make me hopeful my fix solves this issue.

According to Task Manager, Cypress's server was growing to multiple GB in size, so I added some code to make Cypress dump its server's heap memory every 90 seconds. In the contents of that, I found a memory leak and I hope it's the one related to this issue.

Investigation

In the memory dump, I found Cypress was storing hundreds (maybe thousands) copies of the same base64 encoded string in an array named pendingBrowserPreRequests. image

I looked at my company's app and found a css file that has the base64 font file in question hardcoded as the src field. image

Side note: this issue could be remedied on the application side if base64 font files are removed from css files and changed to urls.

Root Cause

Cypress subscribes to the CDP event Network.requestWillBeSent and adds network requests that will be sent to an array. The elements in the array are removed when the corresponding response for the request comes back.

With these fonts inside the URL field of css stylesheets, this triggers Chrome to send a Network.requestWillBeSent event to Cypress with the URL of the request being the entire base64 encoded font file (lol). These requests are not actually going to be sent from the browser as it will parse these values instead, so Cypress never knows when to release these from its array since it never sees a response. It could be argued that this is a Chrome issue, but the fix should be relatively simple on the Cypress side.

I think I can have a PR out tomorrow to fix.

cc: @jennifer-shehane @brian-mann

BlueWinds commented 2 years ago

@DJSdev - Jennifer's currently on leave, so she won't be around to address this, but I'm certainly interested! Please ping me on your PR when you have it ready, I'll make sure it gets the team's attention ASAP.

DJSdev commented 2 years ago

@BlueWinds PR is out!

cypress-bot[bot] commented 2 years ago

The code for this is done in cypress-io/cypress#22460, but has yet to be released. We'll update this issue and reference the changelog when it's released.

cypress-bot[bot] commented 2 years ago

Released in 10.3.0.

This comment thread has been locked. If you are still experiencing this issue after upgrading to Cypress v10.3.0, please open a new issue.