gatsbyjs / gatsby

The best React-based framework with performance, scalability and security built in.
https://www.gatsbyjs.com
MIT License
55.23k stars 10.32k forks source link

JavaScript heap out of memory in the build moment #39103

Open GustavoRosauro opened 1 week ago

GustavoRosauro commented 1 week ago

Preliminary Checks

Description

recently we updated or site from gatsby 4 to 5 but we are facing this error related to JavaScript heap out of memory

after this step during the build success source and transform nodes

I'm receiving this message <--- Last few GCs --->

[2647386:0x709a5f0] 5801810 ms: Scavenge 17474.5 (17912.8) -> 17467.4 (17917.1) MB, 27.80 / 0.00 ms (average mu = 0.970, current mu = 0.977) allocation failure; [2647386:0x709a5f0] 5801880 ms: Scavenge 17478.5 (17917.1) -> 17471.4 (17921.1) MB, 27.58 / 0.00 ms (average mu = 0.970, current mu = 0.977) allocation failure; [2647386:0x709a5f0] 5801991 ms: Scavenge 17482.8 (17921.1) -> 17476.0 (17922.3) MB, 69.62 / 0.00 ms (average mu = 0.970, current mu = 0.977) allocation failure;

<--- JS stacktrace --->

FATAL ERROR: NewSpace::EnsureCurrentCapacity Allocation failed - JavaScript heap out of memory 1: 0xc99960 node::Abort() [/datadrive/vrmobility/agent/_work/_tool/node/20.8.0/x64/bin/node] 2: 0xb6ffcb [/datadrive/vrmobility/agent/_work/_tool/node/20.8.0/x64/bin/node] 3: 0xebe420 v8::Utils::ReportOOMFailure(v8::internal::Isolate, char const, v8::OOMDetails const&) [/datadrive/vrmobility/agent/_work/_tool/node/20.8.0/x64/bin/node] 4: 0xebe707 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate, char const, v8::OOMDetails const&) [/datadrive/vrmobility/agent/_work/_tool/node/20.8.0/x64/bin/node] 5: 0x10d01b5 [/datadrive/vrmobility/agent/_work/_tool/node/20.8.0/x64/bin/node] 6: 0x110f4c9 [/datadrive/vrmobility/agent/_work/_tool/node/20.8.0/x64/bin/node] 7: 0x10e2306 v8::internal::Heap::MarkCompact() [/datadrive/vrmobility/agent/_work/_tool/node/20.8.0/x64/bin/node] 8: 0x10e795d v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::internal::GarbageCollectionReason, char const) [/datadrive/vrmobility/agent/_work/_tool/node/20.8.0/x64/bin/node] 9: 0x10e7e4c v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/datadrive/vrmobility/agent/_work/_tool/node/20.8.0/x64/bin/node] 10: 0x10be151 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/datadrive/vrmobility/agent/_work/_tool/node/20.8.0/x64/bin/node] 11: 0x10bf2e5 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/datadrive/vrmobility/agent/_work/_tool/node/20.8.0/x64/bin/node] 12: 0x109c856 v8::internal::Factory::NewFillerObject(int, v8::internal::AllocationAlignment, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/datadrive/vrmobility/agent/_work/_tool/node/20.8.0/x64/bin/node] 13: 0x14f7831 v8::internal::Runtime_AllocateInOldGeneration(int, unsigned long, v8::internal::Isolate*) [/datadrive/vrmobility/agent/_work/_tool/node/20.8.0/x64/bin/node] 14: 0x74a28bed9ef6 Aborted error Command failed with exit code 134. info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Details about our site, counting the total nodes that we have in gatsby 4 there is aorund 7 millions nodes our consuming memory is around 23 gigas and we already tried to increase the node memory like this NODE_OPTIONS=--max-old-space-size=89152 and we're not even closer of this limit.

So I need some help if someone passed through it when trying to update to the gatsby 5

Reproduction Link

I can't pass the link because this is for a company

Steps to Reproduce

  1. Has a large site in gatsby 4 (more than 7 millions)
  2. Update it to the gatsby 5
  3. trying to build the application "NODE_OPTIONS=--max-old-space-size=89152 gatsby build --prefix-paths --verbose"

Expected Result

we would like to build the application using the gatsby 5 in our local environment

Actual Result

if you have a site the same size than us, you'll receive a message saying FATAL ERROR: NewSpace::EnsureCurrentCapacity Allocation failed - JavaScript heap out of memory

Environment

node 20.8.0
memory: 64 gigas 
disk space: 30 gigas

Config Flags

No response

GustavoRosauro commented 4 days ago

no answers until this moment