search

gatsbyjs / gatsby

The best React-based framework with performance, scalability and security built in.
https://www.gatsbyjs.com
MIT License
55.3k stars 10.31k forks source link

Gatsby 5 Build Fails - FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory #37010

Closed labnol closed 2 years ago

labnol commented 2 years ago

Preliminary Checks

Description

2022-11-10_11-10-20

Reproduction Link

https://gatsby.dev/reproduction

Steps to Reproduce

gatsby build

Expected Result

--

Actual Result

success compile gatsby files - 0.604s
success load gatsby config - 0.261s
success load plugins - 1.462s
success onPreInit - 0.011s
success delete worker cache from previous builds - 0.003s
success initialize cache - 0.144s
success copy gatsby files - 0.266s
success Compiling Gatsby Functions - 0.344s
success onPreBootstrap - 0.363s
success createSchemaCustomization - 0.013s
success Checking for changed pages - 0.002s
success source and transform nodes - 18.700s
info Writing GraphQL type definitions to /Users/amit/google.org/.cache/schema.gql
success building schema - 0.609s
success createPages - 41.528s
success createPagesStatefully - 0.215s
info Total nodes: 15653, SitePage nodes: 4044 (use --verbose for breakdown)
success Checking for changed pages - 0.001s
success onPreExtractQueries - 0.001s
success extract queries from components - 17.224s
success write out redirect data - 0.008s
success Build manifest and related icons - 0.080s
success onPostBootstrap - 0.084s
info bootstrap finished - 85.612s
success write out requires - 0.211s
success Building production JavaScript and CSS bundles - 402.079s
⠦ Building HTML renderer
[============================]   401.665 s 885/885 100% Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs

<--- Last few GCs --->

[44386:0x7fee6404d000]   966679 ms: Scavenge 8035.6 (8219.5) -> 8030.9 (8221.3) MB, 18.3 / 0.0 ms  (average mu = 0.858, current mu = 0.496) allocation failure; 
[44386:0x7fee6404d000]   966709 ms: Scavenge 8037.1 (8221.3) -> 8032.3 (8222.8) MB, 17.7 / 0.0 ms  (average mu = 0.858, current mu = 0.496) allocation failure; 
[44386:0x7fee6404d000]   974510 ms: Scavenge 8038.8 (8222.8) -> 8033.9 (8240.0) MB, 7783.2 / 0.0 ms  (average mu = 0.858, current mu = 0.496) allocation failure; 

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
 1: 0x10d73bd55 node::Abort() (.cold.1) [/Users/amit/n/bin/node]
 2: 0x10c204259 node::Abort() [/Users/amit/n/bin/node]
 3: 0x10c20443e node::OOMErrorHandler(char const*, bool) [/Users/amit/n/bin/node]
 4: 0x10c37f193 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/Users/amit/n/bin/node]
 5: 0x10c547bd5 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/amit/n/bin/node]
 6: 0x10c5465b2 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/amit/n/bin/node]
 7: 0x10c5388ea v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/amit/n/bin/node]
 8: 0x10c539265 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/amit/n/bin/node]
 9: 0x10c51b6de v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/Users/amit/n/bin/node]
10: 0x10c9517ec v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/Users/amit/n/bin/node]
11: 0x10cd4bdf9 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/Users/amit/n/bin/node]
[1]    44385 abort      NODE_OPTIONS=--max-old-space-size=8192 npx gatsby build

Environment

System:
    OS: macOS 13.0
    CPU: (16) x64 Intel(R) Core(TM) i9-9880H CPU @ 2.30GHz
    Shell: 5.8.1 - /bin/zsh
  Binaries:
    Node: 18.12.1 - ~/n/bin/node
    npm: 8.19.3 - ~/.npm-packages/bin/npm
  Browsers:
    Chrome: 107.0.5304.110
    Firefox: 105.0.1
    Safari: 16.1
  npmPackages:
    gatsby: ^5.0.0 => 5.0.0
    gatsby-plugin-canonical-urls: ^5.0.0 => 5.0.0
    gatsby-plugin-catch-links: ^5.0.0 => 5.0.0
    gatsby-plugin-feed: ^5.0.0 => 5.0.0
    gatsby-plugin-google-gtag: ^5.0.0 => 5.0.0
    gatsby-plugin-image: ^3.0.0 => 3.0.0
    gatsby-plugin-manifest: ^5.0.0 => 5.0.0
    gatsby-plugin-mdx: ^5.0.0 => 5.0.0
    gatsby-plugin-no-sourcemaps: ^5.0.0 => 5.0.0
    gatsby-plugin-optimize-svgs: ^1.0.5 => 1.0.5
    gatsby-plugin-postcss: ^6.0.0 => 6.0.0
    gatsby-plugin-sharp: ^5.0.0 => 5.0.0
    gatsby-plugin-sitemap: ^6.0.0 => 6.0.0
    gatsby-plugin-twitter: ^5.0.0 => 5.0.0
    gatsby-remark-autolink-headers: ^6.0.0 => 6.0.0
    gatsby-remark-check-links: ^2.1.0 => 2.1.0
    gatsby-remark-copy-linked-files: ^6.0.0 => 6.0.0
    gatsby-remark-embed-video: ^3.2.1 => 3.2.1
    gatsby-remark-external-links: 0.0.4 => 0.0.4
    gatsby-remark-images: ^7.0.0 => 7.0.0
    gatsby-remark-normalize-paths: ^1.1.0 => 1.1.0
    gatsby-remark-prismjs: ^7.0.0 => 7.0.0
    gatsby-remark-relative-images: ^2.0.2 => 2.0.2
    gatsby-remark-responsive-iframe: ^6.0.0 => 6.0.0
    gatsby-remark-smartypants: ^6.0.0 => 6.0.0
    gatsby-source-filesystem: ^5.0.0 => 5.0.0
    gatsby-transformer-sharp: ^5.0.0 => 5.0.0

Config Flags

No response

amamenko commented 2 years ago

+1 to this

dacevedo12 commented 2 years ago

Likely related to https://github.com/gatsbyjs/gatsby/issues/36899

4.16+ are affected

amamenko commented 2 years ago

Partytown errors show up for me in 4.15.0 as well, 4.16+ won't even build due to OOM issues during the image processing jobs of gatsby-plugin-sharp.

Current usable, error-free version for me is 4.14.1 but, unfortunately, I still do need to upgrade due to MDX incompatibilities such as The new node didn't pass validation: "internal.contentFilePath" is not allowed on Mdx types.

dacevedo12 commented 2 years ago

@amamenko Check https://github.com/gatsbyjs/gatsby/issues/36482#issuecomment-1311974320

hope it helps while this gets fixed

labnol commented 2 years ago

Gatsby v4 works for me without issues. The memory thing happened only after upgraded to Gatsby v5.

LekoArts commented 2 years ago

Currently we only know of https://github.com/gatsbyjs/gatsby/issues/36899, since this issue didn't provide any steps and details on how to reproduce it, I'll close it.

The original issue could also just happen because the system has not enough memory allocated. Hard to tell without details on environment, steps to reproduce, etc. The linked issue is at least confirmed.

labnol commented 2 years ago

Currently we only know of #36899, since this issue didn't provide an steps and details on how to reproduce it, it'll close it.

The original issue could also just happen because the system has not enough memory allocated. Hard to tell without details on environment, steps to reproduce, etc. The linked issue is at least confirmed.

@LekoArts - We moved to Gatsby cloud and the same issue persists. You can view the logs here.

labnol commented 2 years ago

This is related https://github.com/gatsbyjs/gatsby/issues/36899#issuecomment-1319837951

bellons91 commented 1 year ago

Even with the latest Gatsby version (as of today, 5.8.0) the issue is still here.

My blog has 130 articles and 150 images - I don't expect it to generate that much heap allocation.

⠋ Building HTML renderer [============================] 79.136 s 51/51 100% Running gatsby-plugin-sharp.IMAGE_PROCESSING jobs

<--- Last few GCs --->

[25016:000001A5923E6FB0] 168856 ms: Mark-sweep (reduce) 1889.8 (2074.8) -> 1889.7 (2067.5) MB, 731.0 / 0.0 ms (average mu = 0.130, current mu = 0.034) allocation failure; GC in old space requested [25016:000001A5923E6FB0] 169627 ms: Mark-sweep (reduce) 1900.5 (2078.1) -> 1900.3 (2071.1) MB, 693.5 / 0.0 ms (average mu = 0.115, current mu = 0.100) allocation failure; GC in old space requested

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory 1: 00007FF72E092BCF node_api_throw_syntax_error+175519 2: 00007FF72E0183A6 SSL_get_quiet_shutdown+64006 3: 00007FF72E019762 SSL_get_quiet_shutdown+69058 4: 00007FF72EABFDD4 v8::Isolate::ReportExternalAllocationLimitReached+116 5: 00007FF72EAAB132 v8::Isolate::Exit+674 6: 00007FF72E92CF8C v8::internal::EmbedderStackStateScope::ExplicitScopeForTesting+124 7: 00007FF72E93A26D v8::internal::Heap::PublishPendingAllocations+1117 8: 00007FF72E9372F7 v8::internal::Heap::PageFlagsAreConsistent+3367 9: 00007FF72E929A27 v8::internal::Heap::CollectGarbage+2039 10: 00007FF72E9403E3 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath+2099 11: 00007FF72E940C8D v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath+93 12: 00007FF72E9495DA v8::internal::Factory::AllocateRaw+810 13: 00007FF72E95CA1D v8::internal::FactoryBase::NewHeapNumber<0>+685 14: 00007FF72E960EC8 v8::internal::FactoryBase::NewRawTwoByteString+72 15: 00007FF72E708924 v8::internal::String::SlowFlatten+724 16: 00007FF72EAC99ED v8::String::Utf8Length+141 17: 00007FF72E03E521 v8::internal::Malloced::operator delete+18209 18: 00007FF72EA7895D v8::internal::Builtins::code+248237 19: 00007FF72EA78569 v8::internal::Builtins::code+247225 20: 00007FF72EA7882C v8::internal::Builtins::code+247932 21: 00007FF72EA78690 v8::internal::Builtins::code+247520 22: 00007FF72EB5D471 v8::internal::SetupIsolateDelegate::SetupHeap+558449

GustavoRosauro commented 2 months ago

I'm facing the same issue

syehoonkim commented 3 weeks ago

Me too...No bug fix yet?

GustavoRosauro commented 3 weeks ago

I found a way to fix this issue, I've increased the memory descriptors from the agent and everything is working now

vm.max_map_count= new value

syehoonkim commented 3 weeks ago

I found a way to fix this issue, I've increased the memory descriptors from the agent and everything is working now

vm.max_map_count= new value

For Linux users, it must be good news, but I am using Windows...