`Page crashed!`, `net::ERR_INSUFFICIENT_RESOURCES` errors

[mifi]

When running with high concurrency (many browser instances), I'm experiencing these errors after a few seconds of running on Ubuntu:

• fatal Page crashed! error
• net::ERR_INSUFFICIENT_RESOURCES from <FFmpegVideo> <img> error event.

After a lot of digging I found out that it's because the system is running out of disk space. The disk space is being claimed by files that are immediately deleted by chromium (so they don't show up in du -hs) inside the /tmp folder, which has limited space. The files can be seen with lsof:

/tmp/.org.chromium.Chromium.0FNTrV
/tmp/.org.chromium.Chromium.Ra8ezE
/tmp/.org.chromium.Chromium.nQK7nC
/tmp/.org.chromium.Chromium.WQadCm
/tmp/.org.chromium.Chromium.IzapN6
/tmp/.org.chromium.Chromium.Mq2VFV
...

I've tried these options to puppeteer but none seem to affect these tmp files:

PUPPETEER_TMP_DIR
page.setCacheEnabled(false);
--disk-cache-dir
--media-cache-size=0
--disk-cache-size=0

However what turns out to work is to remove the --disable-dev-shm-usage from puppeteer/chromium! If this flag is removed, the files will instead be stored in /dev/shm which has a lot more space (usually many gigabytes on larger AWS EC2 machines). Turns out that the flag was added by default to puppeteer in https://github.com/puppeteer/puppeteer/commit/18f2ecdffdfc70e891750b570bfe8bea5b5ca8c2 - interesting discussion here: https://github.com/puppeteer/puppeteer/issues/1834

Use ignoreDefaultArgs: ['--disable-dev-shm-usage']

See also:

• https://stackoverflow.com/questions/24122506/neterr-insufficient-resources-error-when-adding-numerous-img-elements-to-dom
• https://bugs.chromium.org/p/chromium/issues/detail?id=715363
• https://bugs.chromium.org/p/chromium/issues/detail?id=1085829
• https://bugs.chromium.org/p/chromium/issues/detail?id=333996
• https://bugs.chromium.org/p/chromium/issues/detail?id=265642
• https://bugs.chromium.org/p/chromium/issues/detail?id=736452
• https://github.com/puppeteer/puppeteer/blob/main/docs/troubleshooting.md#tips
• https://github.com/puppeteer/puppeteer/issues/6286
• https://stackoverflow.com/questions/57956697/unhandledpromiserejectionwarning-error-page-crashed-while-using-puppeteer
• https://bugs.chromium.org/p/chromium/issues/detail?id=108055

[ryansch]

@mifi I was debugging the same issue in chromedriver and capybara today and this was where I finally found the answer. Thank you so much for sharing.

[sirmikolai]

@mifi thanks a lot, I had an identical issue when I tried to open new window, which redirected to another system. Then I got error that JS is disabled, but every requests in new tab returned net::ERR_INSUFFICIENT_RESOURCES, so I find solution here.

[caner-cetin]

I am using go-rod, and you literally saved my life.

thank you!

[viktaur]

This seems to indeed get rid of (most of) the net::ERR_INSUFFICIENT_RESOURCES! However, when running inside a Docker container, Puppeteer has trouble finding certain HTML elements, which seems consistent with the recommendation to set the --disable-dev-shm-usage flag for Docker. I wonder if there's a solution that allows for the use of /tmp instead of /dev/shm while reducing the Disk I/O bottleneck.