Docker crashes when renderer process eats up too much memory

[brian-mann]

Related to #348 and #349.

When running headlessly on very long and memory intense applications we are seeing renderer crashes with Docker.

[davidzambrana]

We fixed this problem after implementing a few recommendations we saw here:

• Updated to Cypress 3.8.0
• Moved to this image cypress/browsers:node10.16.0-chrome77
• Started using -headless -b chrome in the run command

• Updated our plugins/index file with

  on('before:browser:launch', (browser = {}, args) => {
  if (browser.name === 'chrome') {
    args.push('--disable-dev-shm-usage')
    return args
  }
  
  return args
  })

The crashes stopped in all the test collections we have (~15)

[tommueller]

I can confirm that the changes @davidzambrana proposed also worked for us so far. We now run into the issue described here https://github.com/cypress-io/cypress/issues/1297 but this is probably more an issue of how we wrote the tests. I will try to fix the tests and keep an eye on this. Thanks @davidzambrana for now!

[mulholo]

To anyone wondering what the --disable-dev-shm-usage flag does, this page is a useful explainer https://developers.google.com/web/tools/puppeteer/troubleshooting#tips

[shlima]

https://github.com/cypress-io/cypress/issues/350#issuecomment-574072211 can confirm the solution, but I've changed cypress/browsers:node10.16.0-chrome77 image to cypress/included:4.0.2 as it currently includes cypress and chrome.

My specs started to run smoothly on Gitlab CI with cypress in docker-in-docker

[adinizs]

We resolved this problem using the image cypress/browsers:node10.16.0-chrome on dockerfile and running chrome headless.

[radDunin]

Hi, i can also confirmed @davidzambrana solution, what is important is that the image should be this one: cypress/browsers:node10.16.0-chrome77 it works even in headed chrome. For now thats doesn't works for me on chrome 80 and chrome 69

[otalandim]

I had the same issue yet I have changed my image from cypress/base to cypress/browsers:node11.13.0-chrome73 and it now works without crashes on gitlab-ci.

image: cypress/browsers:node11.13.0-chrome73

For me it doesnot works

[otalandim]

We fixed this problem after implementing a few recommendations we saw here:

• Updated to Cypress 3.8.0
• Moved to this image cypress/browsers:node10.16.0-chrome77
• Started using -headless -b chrome in the run command
• Updated our plugins/index file with

on('before:browser:launch', (browser = {}, args) => {
    if (browser.name === 'chrome') {
      args.push('--disable-dev-shm-usage')
      return args
    }

    return args
  })

The crashes stopped in all the test collections we have (~15)

For me it doesnot works

Another problem happened...can you help me please

  CypressError: cy.visit() failed trying to load:
 We attempted to make an http request to this URL but the request failed without a response.
 We received this error at the network level:
   > Error: ESOCKETTIMEDOUT
 Common situations why this would fail:
   - you don't have internet access
   - you forgot to run / boot your web server
   - your web server isn't accessible
   - you have weird network configuration settings on your computer
 The stack trace for this error is:
 Error: ESOCKETTIMEDOUT
     at ClientRequest.<anonymous> (/root/.cache/Cypress/4.2.0/Cypress/resources/app/packages/server/node_modules/request/request.js:816:19)
     at Object.onceWrapper (events.js:299:28)
     at ClientRequest.emit (events.js:210:5)
     at TLSSocket.emitRequestTimeout (_http_client.js:690:9)
     at Object.onceWrapper (events.js:299:28)
     at TLSSocket.emit (events.js:210:5)
     at TLSSocket.Socket._onTimeout (net.js:468:8)
     at listOnTimeout (internal/timers.js:531:17)
     at processTimers (internal/timers.js:475:7)
 Because this error occurred during a 'before each' hook we are skipping the remaining tests in the current suite:

[przemuh]

Switching to Chrome with --disable-dev-shm-usage as @davidzambrana suggested - works for me!

Thanks a lot! 🎉

@otalandim Please remember that since Cypress 4.0.0 there is a change in passing arguments to the browser (https://docs.cypress.io/guides/references/migration-guide.html#Plugin-Event-before-browser-launch)

I use GitlabCI. Here are my changes:

GitlabCI config

-  image: cypress/base:12.16.2
+  image: cypress/browsers:node12.16.2-chrome81-ff75

cypress/plugins/index.js

on("before:browser:launch", (browser = {}, launchOptions) => {
  if (browser.name === "chrome") {
    launchOptions.args.push("--disable-dev-shm-usage");
    return launchOptions;
  }
}

run command:

cypress run --browser chrome --headless

[ttomaszewski]

Any idea when this will get resolved for Electron?

[digitalkaoz]

for anyone getting here and want to know how to fix it on gitlab-ci kubernetes executor:

$ kubectl edit cm gitlab-runner-gitlab-runner

config part to add right before # Start the runner in entrypoint see https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2578#note_186055981

    cat >> /home/gitlab-runner/.gitlab-runner/config.toml << EOF
          [[runners.kubernetes.volumes.empty_dir]]
              name = "shm"
              mount_path = "/dev/shm"
              medium = "Memory"
    EOF

[sabarishnarain]

This is still a problem in Azure CI with electron (cypress 4.11) when using the base docker images. The workaround is to use chrome by following @przemuh instructions.

[keul]

For posterity: I've this same issue in Gitlab CI.

Unluckily every gitlab-ci.yml workaround above require the change of the base docker image, while I'm not (and I can't) using any of them: I am manually installing cypress on a Centos (is an internal gitlab installation using a private runner server).

In my case this gitlab comment to issue solved the problem.

[CoryDanielson]

I saw a reduction of 30-50% of memory usage by enabling window.gc() and calling it globally in an afterEach callback.

You can enabled window.gc with browser launch options, or electron command line switches.

// For Chrome, add this flag in plugins
launchOptions.args.push('--js-flags=--expose-gc');

# For Electon, run cypress with this environment variable set
ELECTRON_EXTRA_LAUNCH_ARGS=--js-flags=--expose_gc

plugins/index.js

module.exports = (on, config) => {
    on('before:browser:launch', (browser, launchOptions) => { 
        if (browser.name === 'chrome') {
            launchOptions.args.push('--js-flags=--expose-gc');
        }
        return launchOptions;
    });
};

support/index.js

afterEach(() => {
    cy.window().then(win => {
        if (typeof win.gc === 'function') {
            // calling this more often seems to trigger major GC event more reliably
            win.gc(); win.gc(); win.gc(); win.gc(); win.gc();
        }
    });
});

cypress open peak memory usage - ~50% reduction in peak memory

```console # --js-flags=--expose-gc is set in plugins/index.js cypress open ``` Max memory in MB (recorded by watching Chrome memory tab) | no win.gc | win.gc once | calling win.gc 5 times | | --- | --- | --- | | 535mb | 299 | 267 | | 585mb | 406 | 369 | | 598mb | 372 | 278 | | 570mb | 445 | 298 | | --- | averages | --- | | 572mb | 380 (34% less) | 303 (47% less) |

cypress run peak memory usage - ~30% reduction in peak memory for chrome headless

```console # --js-flags=--expose-gc is set in plugins/index.js cypress run --headless -b chrome ``` (recorded by watching Activity Monitor while tests were running) | Chrome headless, no win.gc()
Chrome Rendered Memory | Chrome headless calling win.gc() 5 times
Chrome Rendered Memory | | --- | --- | | 890 | 580 (~30% less) | | 850 | 560 (~30% less) |

cypress run peak memory usage = ~50% reduction in peak memory for electron

```console ELECTRON_EXTRA_LAUNCH_ARGS=--js-flags=--expose_gc cypress run ``` (recorded by watching Activity Monitor while tests were running) | Electron, no win.gc()
Cypress Helper Renderer Memory | Electron, calling win.gc() 5 times | | --- | --- | | 850mb | 410 (~50% less) | | 750mb | 450 (~50% less) |

Calling window.gc() multiple times was more effective (based on my imperfect tests), I suspect because it possibly caused chrome to trigger a major GC event rather than a minor one.

---

We switched our CI docker run script to use --ipc=host, --disable-dev-shm-usage, chrome headless, and use window.gc() and it's running really well now with no out-of-memory crashes

[phillyc]

For those of you who are trying to solve this issue in GitLab, on the free tier, I found a simple solution.

I was trying to create some smoke tests for my team's application, and noticed that the GitLab CICD runner would typically get through the first 2-3 tests of each test file before crashing with the above out of memory symptoms.

Condense all of the actions and test assertions into 1-2 it('can do foo') test statements.

There appears to be a pretty big memory overhead in the setup/teardown of each of these cases. I was able to take around 60 test cases in my locally passing integration test stack and condense them down into 10 test.js files with 1 test case each.

In summary, try putting all your tests into one test case.

[jackbravo]

What about when running on github actions, like https://github.com/cypress-io/github-action ? How do you enable ipc=host?

[mattvb91]

Started experiencing this again on github actions upgrading from 5.3 to 6.2

[vincentpalita]

Version 6.2.1 seems to fix that for me.

[mattvb91]

Unfortunately 6.2.1 still crashing https://github.com/corejam/corejam/runs/1650892998?check_suite_focus=true#step:9:290 staying on 5.3 for now. Maybe someone else can chime in a version number when it started happening for them between 5.3 & 6.2 as I dont have the time to manually go through each version / build the associated docker images

[mattvb91]

https://github.com/corejam/corejam/runs/1784204472?check_suite_focus=true#step:9:313

Still no luck on 6.3.0

5.3.0 still stable for us have not had any chrome crashes there

---

Edit crashes on 6.4.0 too: https://github.com/corejam/corejam/runs/1820965122?check_suite_focus=true#step:9:287

[mattvb91]

Anyone looking to enable IPC=host on github actions it looks like it is possible like this: https://github.com/corejam/corejam/pull/311/files#diff-014228303dff9a1af15f4bbd18401f906380129b10ae2a2c62f8b8be592ff88eR105

You can see it setting the option here: https://github.com/corejam/corejam/runs/1888917637?check_suite_focus=true#step:2:111 /usr/bin/docker create --name 01f73992fe8148a89514be690697b85d_corejamcypress640_aa44b9 --label 5588e4 --workdir /__w/corejam/corejam --network github_network_3bce8c6715944b47a995b42f42973efa --ipc=host ...

More context here: https://github.com/actions/virtual-environments/discussions/2697#discussioncomment-362677

Just let it run through 3 times on our setup and its finally running it seems!!

Can anyone else confirm? If its working for others ill put in a MR with the above sample for docs

Edit: Have since put on about 10 runs from merge requests and its stable!

[dobriai]

FWIW, in our case adding --shm-size=512m seems to have quieted down the issue. For now. Using cypress@6.4.0.

[TimothyRHuertas]

If you added --ipc=host and you still get the error, try restarting docker 🤦

[cypress-bot[bot]]

The code for this is done in cypress-io/cypress#15825, but has yet to be released. We'll update this issue and reference the changelog when it's released.

[flotwig]

In 7.0.1, Cypress passes disable-dev-shm-usage to Chrome and Electron on startup, which should remove the need to use --ipc=host or increase the /dev/shm size. Once it's released this should be solved.

[cypress-bot[bot]]

Released in 7.0.1.

This comment thread has been locked. If you are still experiencing this issue after upgrading to Cypress v7.0.1, please open a new issue.