Memory leak whilst generating image thumbnails in a Circle CI Build

[RamonBalthazar]

Description

In CircleCI during the build process while generating image thumbnails, the console "freezes" and after some time a memory leak message is displayed:

UNHANDLED EXCEPTION spawn ENOMEM

My main suspicion is that something is not respecting the GATSBY_CPU_COUNT=2 environment variable so the workers can't balance themselves, probably Image Sharp?

Steps to reproduce

Clear steps describing how to reproduce the issue. Please please please link to a demo project if possible, this makes your issue much easier to diagnose (seriously).

The issue is happening in CircleCI for a private repo, any idea on what and how I could share more information about it?

Expected result

The build shouldn't be crashing.

We do have loads of images and some are pretty heavy, but I don't think the build should ever crash, only take longer to process them all. At least that's what happens locally and that's what I expect to be happening in CircleCI.

Actual result

It starts generating the thumbnails but fails right in the middle of it due to a memory leak.

success open and validate gatsby-configs — 1.070
success load plugins — 1.161
success onPreInit — 0.017
success delete html and css files from previous builds — 0.019
success initialize cache — 0.022
success copy gatsby files — 0.059
success onPreBootstrap — 0.027
success source and transform nodes — 3.282
success building schema — 0.275
success createPages — 2.692
success createPagesStatefully — 0.097
success onPreExtractQueries — 0.021
success update schema — 40.457
success extract queries from components — 0.573
success write out requires — 0.066
success write out redirect data — 0.062
success Build manifest and related icons — 0.060
success onPostBootstrap — 0.107

info bootstrap finished - 51.925 s

*** LOADS OF LOADING MESSAGES, WHICH I REDACTED, see pic below ***
WARNING: We noticed you're using the `useBuiltIns` option without declaring a core-js version. Currently, we assume version 2.x when no version is passed. Since this default version will likely change in future versions of Babel, we recommend explicitly setting the core-js version you are using via the `corejs` option.

You should also be sure that the version you pass to the `corejs` option matches the version specified in your `package.json`'s `dependencies` section. If it doesn't, you need to run one of the following commands:

  npm install --save core-js@2    npm install --save core-js@3
  yarn add core-js@2              yarn add core-js@3

error spawn ENOMEM

UNHANDLED REJECTION spawn ENOMEM

See our docs page for more info on this error: https://gatsby.dev/issue-how-to

  Error: spawn ENOMEM

  - child_process.js:366 ChildProcess.spawn
    internal/child_process.js:366:11

  - index.js:204 module.exports
    [code]/[imagemin-pngquant]/[execa]/index.js:204:26

  - index.js:52 input
    [code]/[imagemin-pngquant]/index.js:52:13

  - index.js:71 Function.module.exports.buffer
    [code]/[imagemin]/index.js:71:31

  - process-file.js:198 pipeline.toBuffer.then.sharpBuffer
    [code]/[gatsby-plugin-sharp]/process-file.js:198:105

error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
Exited with code 1

Environment

The environment in CircleCI:

  System:
    OS: Linux 4.15 Debian GNU/Linux 9 (stretch) 9 (stretch)
    CPU: (36) x64 Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
    Shell: 4.4.12 - /bin/bash
  Binaries:
    Node: 10.16.3 - /tmp/yarn--1567168012210-0.10582705648443946/node
    Yarn: 1.17.3 - /tmp/yarn--1567168012210-0.10582705648443946/yarn
    npm: 6.9.0 - /usr/local/bin/npm
  Languages:
    Python: 2.7.13 - /usr/bin/python
  npmPackages:
    gatsby: ^2.15.0 => 2.15.0 
    gatsby-image: ^2.2.15 => 2.2.15 
    gatsby-plugin-favicon: ^3.1.6 => 3.1.6 
    gatsby-plugin-manifest: ^2.2.11 => 2.2.11 
    gatsby-plugin-offline: ^2.2.10 => 2.2.10 
    gatsby-plugin-react-helmet: ^3.1.5 => 3.1.5 
    gatsby-plugin-sharp: ^2.2.18 => 2.2.18 
    gatsby-plugin-styled-components: ^3.1.3 => 3.1.3 
    gatsby-plugin-typescript: ^2.1.5 => 2.1.5 
    gatsby-source-filesystem: ^2.1.18 => 2.1.18 
    gatsby-source-graphql: ^2.1.11 => 2.1.11 
    gatsby-transformer-sharp: ^2.2.12 => 2.2.12 

30 AUG UPDATE: Updated all gatsby packages, the problem persists

[ArthurHwang]

I am running into the exact same issue. I tried adding NODE_OPTIONS=--max_old_space_size=4096 and messing with GATSBY_CPU_COUNT but no luck.

[pierrenel]

Same over here :( Netlify is taking eons to do my images too (over 15 minutes) so I figured I'd go the CircleCI -> Firebase route.

[gusliedke]

Can we skip Generating image thumbnails — 4454/4454 - 1130.272 s?

Is thumbnails here = all images?

It should regenerate only the modified ones.

[LekoArts]

It should regenerate only the modified ones.

Since CircleCI doesn't keep the .cache directory around (where the images are stored during consecutive builds e.g. locally) you'll re-generate all images.

[vpicone]

Was getting this exact issue, turns out we had a accidentally had a 4.5mb png that, when processed by sharp, pushed the memory limits of the circleci “free” tier. Scaling the image down solved the problem.

[gatsbot[bot]]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 30 days of inactivity. It’s been at least 20 days since the last update here.

If we missed this issue or if you want to keep it open, please reply here. You can also add the label "not stale" to keep this issue open!

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[voltidev]

I have the same issue on CircleCI.

...

 [] 810/1462 136.8 s 55% Generating image thumbnails [] 812/1462 136.8 s 55% Generating image thumbnails [] 814/1462 136.8 s 55% Generating image thumbnails
 [] 816/1462 142.1 s 55% Generating image thumbnails [] 836/1462 142.1 s 57% Generating image thumbnails
 [] 837/1462 142.3 s 57% Generating image thumbnails [] 841/1462 142.5 s 57% Generating image thumbnails
error UNHANDLED EXCEPTION spawn ENOMEM

  Error: spawn ENOMEM

  - child_process.js:366 ChildProcess.spawn
    internal/child_process.js:366:11

...

[gatsbot[bot]]

Hey again!

It’s been 30 days since anything happened on this issue, so our friendly neighborhood robot (that’s me!) is going to close it.

Please keep in mind that I’m only a robot, so if I’ve closed this issue in error, I’m HUMAN_EMOTION_SORRY. Please feel free to reopen this issue or create a new one if you need anything else.

As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks again for being part of the Gatsby community!

[FONDSMATIVE]

Hi! Thank you so much for this great gatsby community! I really love it!

We do have a small project using Gatsby, of course! We do use CircleCI as well, and we have exactly the same problems described here. I will attach a screenshot as well.

I think, handling large images should not be a problem at all - to run into spawn ENOMEM is really a very tough questions. Isn't it @LekoArts ?

It would be great to resolve those problems, a lot of us are experiencing. Many many thanks!

P.S. Here we go...

[leonfs]

If the problem is in libvips, then perhaps setting this environment variable to 1? https://github.com/libvips/libvips/blob/master/libvips/iofuncs/threadpool.c#L304

libvips/iofuncs/threadpool.c:304

• set by the environment variable VIPS_CONCURRENCY, or if that is not set, the https://github.com/libvips/libvips|libvips/libvipslibvips/libvips | Added by GitHub

$ node sharp.js
8
$ VIPS_CONCURRENCY=1 node sharp.js
1

where sharp.js contains:

const sharp = require('sharp');
console.log(sharp.concurrency());

We were using a small resource (1 core) in CircleCI which meant that having 8 workers caused sporadic not enough memory issues. We saw built times improvements after this change too, likely to less resource contention.

I hope it helps people facing this issue in the future and avoids unnecessary bigger build machines.

Credits to @dylansmith