FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory when trying to run `one-shot.js`

[deepaerial]

When trying to run one-shot.js file from examples folder I get following error:

<--- Last few GCs --->

[68402:0x7fba84200000]    98314 ms: Mark-Compact 4038.0 (4144.9) -> 4025.8 (4141.9) MB, 1518.83 / 0.01 ms  (average mu = 0.508, current mu = 0.241) allocation failure; GC in old space requested
[68402:0x7fba84200000]   101303 ms: Mark-Compact 4035.5 (4143.9) -> 4030.9 (4145.4) MB, 2642.13 / 0.01 ms  (average mu = 0.308, current mu = 0.116) allocation failure; GC in old space requested

<--- JS stacktrace --->

FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
----- Native stack trace -----

 1: 0x106bfece0 node::OOMErrorHandler(char const*, v8::OOMDetails const&) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
 2: 0x106db626c v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, v8::OOMDetails const&) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
 3: 0x106fa6007 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
 4: 0x106fa4869 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
 5: 0x106f990d1 v8::internal::HeapAllocator::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
 6: 0x106f99b15 v8::internal::HeapAllocator::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
 7: 0x106f7b46c v8::internal::Factory::AllocateRaw(int, v8::internal::AllocationType, v8::internal::AllocationAlignment) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
 8: 0x106f6f4ab v8::internal::FactoryBase<v8::internal::Factory>::NewBytecodeArray(int, unsigned char const*, int, int, v8::internal::Handle<v8::internal::FixedArray>) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
 9: 0x10709e68c v8::internal::Handle<v8::internal::BytecodeArray> v8::internal::interpreter::BytecodeArrayWriter::ToBytecodeArray<v8::internal::Isolate>(v8::internal::Isolate*, int, int, v8::internal::Handle<v8::internal::ByteArray>) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
10: 0x1070a0d95 v8::internal::Handle<v8::internal::BytecodeArray> v8::internal::interpreter::BytecodeGenerator::FinalizeBytecode<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Script>) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
11: 0x1070c55b5 v8::internal::CompilationJob::Status v8::internal::interpreter::InterpreterCompilationJob::DoFinalizeJobImpl<v8::internal::Isolate>(v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Isolate*) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
12: 0x1070c542c v8::internal::interpreter::InterpreterCompilationJob::FinalizeJobImpl(v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Isolate*) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
13: 0x106e7a53d v8::internal::CompilationJob::Status v8::internal::(anonymous namespace)::FinalizeSingleUnoptimizedCompilationJob<v8::internal::Isolate>(v8::internal::UnoptimizedCompilationJob*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Isolate*, std::__1::vector<v8::internal::FinalizeUnoptimizedCompilationData, std::__1::allocator<v8::internal::FinalizeUnoptimizedCompilationData>>*) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
14: 0x106e730c5 bool v8::internal::(anonymous namespace)::IterativelyExecuteAndFinalizeUnoptimizedCompilationJobs<v8::internal::Isolate>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Handle<v8::internal::Script>, v8::internal::ParseInfo*, v8::internal::AccountingAllocator*, v8::internal::IsCompiledScope*, std::__1::vector<v8::internal::FinalizeUnoptimizedCompilationData, std::__1::allocator<v8::internal::FinalizeUnoptimizedCompilationData>>*, std::__1::vector<v8::internal::DeferredFinalizationJobData, std::__1::allocator<v8::internal::DeferredFinalizationJobData>>*) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
15: 0x106e72ba4 v8::internal::Compiler::Compile(v8::internal::Isolate*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*, v8::internal::CreateSourcePositions) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
16: 0x106e73558 v8::internal::Compiler::Compile(v8::internal::Isolate*, v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag, v8::internal::IsCompiledScope*) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
17: 0x1073f0b66 v8::internal::Runtime_CompileLazy(int, unsigned long*, v8::internal::Isolate*) [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
18: 0x1077cadb6 Builtins_CEntry_Return1_ArgvOnStack_NoBuiltinExit [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
19: 0x10773dba4 Builtins_CompileLazy [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
20: 0x10773c79c Builtins_InterpreterEntryTrampoline [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
21: 0x10773c79c Builtins_InterpreterEntryTrampoline [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
22: 0x10773c79c Builtins_InterpreterEntryTrampoline [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
23: 0x10773c79c Builtins_InterpreterEntryTrampoline [/Users/myuser/.nvm/versions/node/v20.14.0/bin/node]
[1]    68402 abort      node one-shot.js

Node version: v20.14.0

[YunzheZJU]

This seems to be an issue of the env you are using to execute the javascript code.

Have you tried running node with NODE_OPTIONS=--max_old_space_size=4096? This should raise the heap limit to avoid this kind of error.

https://stackoverflow.com/questions/38558989/node-js-heap-out-of-memory

[s123121]

I am pretty sure this got into the endless loop of calling while to generate the poToken, it can not generate the valid poTken -> memory leak

[YunzheZJU]

I have switched to node-v20.14.0-darwin-arm64 and manually run the endless loop in https://github.com/YunzheZJU/youtube-po-token-generator/blob/main/lib/task.js.

The program does not crash even if it has looped for 200+ times.

The memory usage of node keeps around 1.0GB ~ 1.2GB after some loops, while it was 500+ MB in the beginning.

However, when I switch to node 20.1.0, that decreases from >500MB to <200MB and never reached 300MB even if 200 loops is run.

So I thick that there should be no memory leak caused by myself (at least before 200 loops are reached) but there is a chance that logics from youtube's code may introduce memory leaks when it cannot generate the valid token.

And the node version may affect the memory usage, as you can see, so switching node versions may do some help.

---

Strangely it would output extra long poTokens in the 62th, 123th and 182th loop, which consumes a lot of time before the 63th, 124th, and 183th ones are generated.