search

gatsbyjs / gatsby

The best React-based framework with performance, scalability and security built in.
https://www.gatsbyjs.com
MIT License
55.27k stars 10.31k forks source link

Gatsby@2.11.0 Build failure (JavaScript heap out of memory) #15190

Closed moonmeister closed 5 years ago

moonmeister commented 5 years ago

Description

Gatsby 2.11 breaks site build. I'm guessing it's the new functionality around building node_modules as well as the site (#14111).

i was updating dependencies on master when build started breaking. I've updated to 2.10.5 on an update branch with no troubles but as soon as I move Gatsby to 2.11 the build breaks.

Steps to reproduce

  1. Clone master or update branches from https://github.com/moonmeister/moonmeister.net
  2. update gatsby dep to gatsby@2.11.0
  3. run build => see breakage.

Expected result

build should be successful.

Actual result

[BABEL] Note: The code generator has deoptimised the styling of /home/ajmoon/code/moonmeister/moonmeister.net/node_modules/netlify-cms-app/dist/netlify-cms-app.js as it
⠦ Building production JavaScript and CSS bundles

<--- Last few GCs --->

[19712:0x2dbca30]    45370 ms: Scavenge 1338.2 (1423.9) -> 1337.3 (1423.9) MB, 2.9 / 0.0 ms  (average mu = 0.163, current mu = 0.102) allocation failure 
[19712:0x2dbca30]    45374 ms: Scavenge 1338.4 (1423.9) -> 1337.5 (1423.9) MB, 2.8 / 0.0 ms  (average mu = 0.163, current mu = 0.102) allocation failure 
[19712:0x2dbca30]    45378 ms: Scavenge 1338.6 (1423.9) -> 1337.7 (1424.4) MB, 2.6 / 0.0 ms  (average mu = 0.163, current mu = 0.102) allocation failure 

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 0x34eb54adbe1d]
    1: StubFrame [pc: 0x34eb54a875c2]
Security context: 0x2caa7a21e6e9 <JSObject>
    2: /* anonymous */(aka /* anonymous */) [0x23804dd52ac9] [/home/ajmoon/code/moonmeister/moonmeister.net/node_modules/@babel/core/lib/transformation/file/merge-map.js:~155] [pc=0x34eb5534b963](this=0x01389b5022b1 <null>,m=0x1c2e6adbae29 <Object map = 0x11c6eb590b11>)
    3: arguments adaptor frame: 3->1
    4: forEach...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8dc510 node::Abort() [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 2: 0x8dc55c  [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 3: 0xad9b5e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 4: 0xad9d94 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 5: 0xec7bf2  [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 6: 0xec7cf8 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 7: 0xed3dd2 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 8: 0xed4704 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
 9: 0xed7371 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
10: 0xea07f4 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationSpace) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
11: 0x114018e v8::internal::Runtime_AllocateInNewSpace(int, v8::internal::Object**, v8::internal::Isolate*) [/home/ajmoon/.nvm/versions/node/v10.15.3/bin/node]
12: 0x34eb54adbe1d 
Aborted (core dumped)
error Command failed with exit code 134.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Environment

  System:
    OS: Linux 5.0 Ubuntu 19.04 (Disco Dingo)
    CPU: (8) x64 Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
    Shell: 5.0.3 - /bin/bash
  Binaries:
    Node: 10.15.3 - ~/.nvm/versions/node/v10.15.3/bin/node
    Yarn: 1.15.2 - ~/.nvm/versions/node/v10.15.3/bin/yarn
    npm: 6.9.0 - ~/.nvm/versions/node/v10.15.3/bin/npm
  Browsers:
    Chrome: 75.0.3770.100
    Firefox: 67.0.4
  npmPackages:
    gatsby: 2.11.0 => 2.11.0 
    gatsby-image: ^2.0.34 => 2.2.3 
    gatsby-plugin-eslint: ^2.0.4 => 2.0.5 
    gatsby-plugin-manifest: ^2.0.24 => 2.2.0 
    gatsby-plugin-netlify: ^2.0.13 => 2.1.0 
    gatsby-plugin-netlify-cms: ^4.1.0 => 4.1.0 
    gatsby-plugin-offline: ^2.0.25 => 2.2.0 
    gatsby-plugin-react-helmet: ^3.0.10 => 3.1.0 
    gatsby-plugin-robots-txt: ^1.4.0 => 1.4.0 
    gatsby-plugin-sass: ^2.0.11 => 2.1.0 
    gatsby-plugin-sharp: ^2.0.30 => 2.2.1 
    gatsby-plugin-sitemap: ^2.0.11 => 2.2.0 
    gatsby-remark-images: ^3.0.10 => 3.1.2 
    gatsby-remark-relative-images: ^0.2.2 => 0.2.2 
    gatsby-source-filesystem: ^2.0.27 => 2.1.1 
    gatsby-transformer-remark: ^2.3.8 => 2.5.0 
    gatsby-transformer-sharp: ^2.1.17 => 2.2.0 
  npmGlobalPackages:
    gatsby-cli: 2.7.2
    gatsby-dev-cli: 2.5.0
moonmeister commented 5 years ago

Okay, so this is definitely a node issue. It seems that the extra transpiling being done is just now making node hit it's default limit in my case. I ran export NODE_OPTIONS=--max_old_space_size=4096 to expand the defaults and build is now happy.

this seems strange that node doesn't just request more memory if it's available, also I'm a little concerned how this might affect folks build servers.

moonmeister commented 5 years ago

Yeah, affects Netlify too. I was able to get Netlify to build correctly by adding an environment variable:

[build.environment]
  NODE_OPTIONS = "--max_old_space_size=4096"
moonmeister commented 5 years ago

This seems like it might negatively affect a lot of people. Any way we can make this more memory efficient?

vickylance commented 5 years ago

@moonmeister OMG thank you so much I just updated my package.json file and suddenly this issue crept up. And yes increasing the size solved the issue.

moonmeister commented 5 years ago

@vickylance figured I wouldn't be the only one. Glad you figured it out.

vickylance commented 5 years ago

However netlify does not seem to build even after increasing the node size. Its not throwing an error but its just staying building forever.

EDIT: it got published. But it took a lot of time to build

Zellement commented 5 years ago

I've got the same issue I think, @vickylance and @moonmeister - I can test the environment variable fix if someone can guide me how? :) Not done them before - is it just as here in Netlify?

moonmeister commented 5 years ago

Not sure you need the quotes but that looks correct.

Zellement commented 5 years ago

OK - didn't work with the quotes initially. I'll try without.

EDIT: Works without quotes in Netlify

moonmeister commented 5 years ago

Yeah, build times have definitely increased. I'm not sure what value this adds, I'm sure it does but it seems to have some significant downsides too.

vickylance commented 5 years ago

Add in netlify like this.

Screen Shot 2019-06-28 at 2 26 47 AM
vickylance commented 5 years ago

@moonmeister Yes my build time was less than a min and now its more than 6 mins

Zellement commented 5 years ago

Same here; environment variable fixed it, but it took an age to deploy.

Not sure about you guys but I also got a lot of lines of...

10:04:46 PM: Pkg: FLAGS_MISMATCH

Not seen that before myself.

EDIT: In Netlify EDIT 2: Gone from 4 mins to 7 mins.

moonmeister commented 5 years ago

@gatsbyjs/core Can we get some input here? This isn't strictly a Gatsby issue but definitely caused by a decision the Gatsby project made. IS there a way to disable this for folks that don't want it, or maybe a way to optimize the process to increase performance/build times (or at least not break node)?

wardpeet commented 5 years ago

It only took 20-40s longer on gatsbyjs.org when transpiling node_modules so it's weird that it's so heavy for your sites. I'll have a look at your repro! @moonmeister

moonmeister commented 5 years ago

@wardpeet Thanks! My site is only page (aside from the cms), so it wasn't much to start with...but it probably still doubled.

If anyone else has repositories they can share that may be helpful.

homearanya commented 5 years ago

Same problem on my side. This has been driving me bananas. I'm glad to find out it's an issue on Gatsby's last version

wardpeet commented 5 years ago

Seems like gatsby-netlify-cms being the issue. I'll cook up a fix.

Sorry for the inconveniences

wardpeet commented 5 years ago

https://github.com/gatsbyjs/gatsby/pull/15191 seems to be ready to get merged!

moonmeister commented 5 years ago

@wardpeet can you explain why this plugin specifically is an issue? Is it purely its size?

erquhart commented 5 years ago

gatsby-plugin-netlify-cms runs an independent Webpack build within Gatsby's Webpack build (insert "yo dawg" reference here) using a modified version of the Gatsby Webpack config. Gatsby 2.11.0 no longer excludes node_modules, which causes the plugin's Webpack build to attempt processing the Netlify CMS modules with Babel. The modules are massive and also prebuilt, so it will (and should) crash your system.

The PR I raised just adds that exclusion back in to the plugin's Webpack config. Please don't override your node max file size limit to fix this!

bscs-dev-team commented 5 years ago

Hi all, this had us stumped for a minute.

Thanks, @moonmeister for posting a temporary fix.

Yes, it increased our build time quite a bit too but at least it is building again.

@erquhart - are you suggesting we shouldn't use the temp fix?

erquhart commented 5 years ago

Sorry, no - definitely good as a temp fix, just recommending folks revert back once a proper fix is released.

JelaMiraj commented 5 years ago

Will #15191 be sufficient for a developer who uses gatsby-plugin-netlify-cms & uses Yarn's PnP feature?

hendra-go commented 5 years ago

I also experienced the same thing, high memory usage on gatsby build.

after investigated a little further, i think this block of code from https://github.com/gatsbyjs/gatsby/pull/14111/commits/4210885b605f61d49fec786509441bb28dc0570e commit that caused the problem:

    let js = ({ exclude = [], ...options } = {}) => {
      const excludeRegex = [
        `core-js|event-source-polyfill|webpack-hot-middleware/client`,
      ].concat(exclude)

      return {
        test: /\.(js|mjs|jsx)$/,
        exclude: new RegExp(excludeRegex.join(`|`)),
        type: `javascript/auto`,
        use: [loaders.js(options)],
      }

I succesfully running gatsby build after revert webpack-utils.js back to this https://github.com/gatsbyjs/gatsby/pull/14111/commits/d46f17b8fe7a9eb566f257b0c8174ae273660875 commit

  {
    let js = (options = {}) => {
      return {
        test: /\.(js|mjs|jsx)$/,
        exclude: /core-js|event-source-polyfill|webpack-hot-middleware\/client/,
        type: `javascript/auto`,
        use: [loaders.js(options)],
      }
    }

    rules.js = js
  }

I suspect the RegExp causing this memory inefficiency

erquhart commented 5 years ago

@hendra-go is the project you're seeing this issue in using the Netlify CMS plug-in?

Sent with GitHawk

hendra-go commented 5 years ago

@hendra-go is the project you're seeing this issue in using the Netlify CMS plug-in?

@erquhart No, i dont use Netlify CMS plug-in, i try to build locally, using gatsby@2.11.0 even using node --max-old-space-size=4096 when running gatsby build did not work on my laptop with 8GB memory.

erquhart commented 5 years ago

Interesting, maybe Gatsby itself is having the same issues the Netlify CMS plug-in build is having. It's likely due to the commit you mentioned (from #14111), but I don't think it's a RegExp perf issue - seems more probable that it's due to Babel now processing modules that used to be excluded, especially in node_modules. I could easily be wrong though, @wardpeet should have more insight.

Sent with GitHawk

ssijak commented 5 years ago

I am too experiencing this problem with Gatsby 2.11.x versions and not with 2.10. I downgraded for now.

SpicyPete commented 5 years ago

Also getting this issue with Gatsby 2.11.x while using styled-components. Sticking with 2.10 for now.

hupe1980 commented 5 years ago

I have the same issue. A downgrade to 2.10 solved the problem

wardpeet commented 5 years ago

@hendra-go mind sharing your repo?

wardpeet commented 5 years ago

new version of gatsby-netlify-cms got published 4.1.1, it should fix this issue.

pogo19 commented 5 years ago

It seems fixed for me. Thanks!

SpicyPete commented 5 years ago

Still broken for me when using styled-components 🤔

rwieruch commented 5 years ago

I ran into this today as well:

[BABEL] Note: The code generator has deoptimised the styling of undefined 
as it exceeds the max of 500KB.

It happens for a large markdown file (exactly this long read blog post which I want to transfer from my old Hugo website to my new Gatsby website). In Gatsby I use PrismJS and MDX. The 1600th line of markdown kills it.

So I am not entirely sure whether it is related, because I am not using Netlify. If not related, I am happy to open a new issue.

My dependencies:

  "dependencies": {
    "@mdx-js/mdx": "^1.0.21",
    "@mdx-js/react": "^1.0.21",
    "axios": "^0.19.0",
    "disqus-react": "^1.0.6",
    "dotenv": "^7.0.0",
    "gatsby": "^2.12.0",
    "gatsby-image": "^2.2.3",
    "gatsby-link": "^2.2.0",
    "gatsby-mdx": "^0.6.3",
    "gatsby-plugin-catch-links": "^2.1.0",
    "gatsby-plugin-feed": "^2.3.1",
    "gatsby-plugin-google-analytics": "^2.1.1",
    "gatsby-plugin-manifest": "^2.2.0",
    "gatsby-plugin-offline": "^2.2.0",
    "gatsby-plugin-react-helmet": "^3.1.0",
    "gatsby-plugin-sentry": "^1.0.1",
    "gatsby-plugin-sharp": "^2.2.1",
    "gatsby-plugin-sitemap": "^2.2.1",
    "gatsby-plugin-styled-components": "^3.1.0",
    "gatsby-remark-autolink-headers": "^2.1.0",
    "gatsby-remark-copy-linked-files": "^2.1.0",
    "gatsby-remark-images": "^3.1.2",
    "gatsby-remark-prismjs": "^3.3.0",
    "gatsby-source-filesystem": "^2.1.1",
    "gatsby-transformer-remark": "^2.5.0",
    "gatsby-transformer-sharp": "^2.2.0",
    "prismjs": "^1.16.0",
    "react": "next",
    "react-dom": "next",
    "react-facebook-pixel": "^0.1.1",
    "react-helmet": "~5.2.1",
    "react-quora-pixel": "0.0.5",
    "react-twitter-conversion-tracker": "^1.0.0",
    "react-youtube": "^7.9.0",
    "styled-components": "^4.3.2"
  },

Before I had Gatsby 2.3.25 and ran into the same issue (in case we want to exclude a regression bug). Then I updated full of hope that a new version would resolve it.

SpicyPete commented 5 years ago

Gatsby 2.12.x seemed to of fixed all of my issues

wernerglinka commented 5 years ago

I am following the Gatsby tutorial from https://www.netlifycms.org/docs/gatsby/ I am running Gatsby 2.13.2, build on Netlify crashed but the fix NODE_OPTIONS = "--max_old_space_size=4096" did the job.

moonmeister commented 5 years ago

@wernerglinka netlify-cms changed the name of it's package to netlify-cms-app. That may be affecting you, not sure.

jamesgrubb commented 5 years ago

Having same issue here.

erquhart commented 5 years ago

This was fixed - are you sure it's the same exact issue? Also make sure you have the latest version of both Gatsby and the Netlify CMS Gatsby plugin.

Sent with GitHawk

wernerglinka commented 5 years ago

Thanks Alex, I updated to netlify-cms-app and everything works. Werner

Sent from my iPad

On Jul 4, 2019, at 19:20, Alex Moon notifications@github.com wrote:

@wernerglinka netlify-cms changed the name of it's package to netlify-cms-app. That may be affecting you, not sure.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

jamesgrubb commented 5 years ago

I added the environment variable --max-old-space-size=4096 to netlify and managed a successful build. I deleted lock files deleted node_modules and re-installed. Same issue as far as I can `see.

"dependencies": {
    "gatsby": "^2.13.4",
    "gatsby-plugin-netlify-cms": "^4.1.1",
    "netlify-cms-app": "^2.9.5",
    "react": "^16.8.6",
    "react-dom": "^16.8.6"
  }
erquhart commented 5 years ago

Can you help us reproduce? Repo we can clone?

Sent with GitHawk

rdela commented 5 years ago

In case it helps figure any of this out, this repo that uses Netlify CMS appears to still be broken under 2.13.6 2.13.9, but everything is fine under 2.10.5. See master branch vs https://github.com/rdela/voidcluster/pull/54; build failure log at https://app.netlify.com/sites/voidcluster/deploys/5d20f59e9f191b000a971e08 https://app.netlify.com/sites/voidcluster/deploys/5d242cc7332e50bd7113ca72

@erquhart I commented on your PR already on Jul 6, 2019 at 12:40 PM PDT, a few hours after your last post here: https://github.com/gatsbyjs/gatsby/pull/15191#issuecomment-508949236 ...not sure whether you got notified already or not, if so, sorry for the double mention and thanks for all the work trying to resolve this!

ethan-hurst commented 5 years ago

Yeah, affects Netlify too. I was able to get Netlify to build correctly by adding an environment variable:

[build.environment]
  NODE_OPTIONS = "--max_old_space_size=4096"

This worked for me <3

rdela commented 5 years ago

Added "(JavaScript heap out of memory)" to end of issue title just now (in addition to reopening yesterday) to try and prevent more duplicate issues like #15540

wardpeet commented 5 years ago

I'll be closing this one in favor of #15256

Please move the discussion to that thread. I have created a PR that hopefully fixes the OOM issue https://github.com/gatsbyjs/gatsby/pull/15588

rdela commented 5 years ago

#15588 #15591 has done the trick for me thanks @wardpeet!

vientang commented 5 years ago

I just got hit with this. Gatsby version 2.13.31, Node 10.15.3 via npx and npm 6.4.1

I had imported a file that was 8mb, which was working just a few days ago. @moonmeister 's solution worked for me - increase memory with export NODE_OPTIONS=--max_old_space_size=4096