search

josdejong / workerpool

Offload tasks to a pool of workers on node.js and in the browser
Apache License 2.0
2.06k stars 147 forks source link

Main node process crashes with OOM error #378

Closed VP-0822 closed 1 year ago

VP-0822 commented 1 year ago

Switching workerpool from child_process to worker thread leads to OOM error. Currently, there is only one worker configured and only one task in the queue. After a worker has initialized, any DB connection-related activity (createConnection, exec, or even rollback) is performed in the main thread, the main node process crashes with an error,

FATAL ERROR: v8::HandleScope::CreateHandle() Cannot create a handle without a HandleScope
 1: 0x10268dc84 node::Abort() [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
 2: 0x10268dd7c node::OOMErrorHandler(char const*, bool) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
 3: 0x1027d28fc v8::Utils::ReportApiFailure(char const*, char const*) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
 4: 0x102915764 v8::internal::HandleScope::Extend(v8::internal::Isolate*) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
 5: 0x1027d49e4 v8::EscapableHandleScope::EscapableHandleScope(v8::Isolate*) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
 6: 0x102654114 napi_open_escapable_handle_scope [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
 7: 0x11be6402c Napi::EscapableHandleScope::EscapableHandleScope(Napi::Env) [/Users/VP-0822/Documents/git/bookshop/node_modules/@sap/hana-client/prebuilt/darwinarm64-xcode12/hana-client.node]
 8: 0x11be5b888 Napi::FunctionReference::Call(std::initializer_list<napi_value__*> const&) const [/Users/VP-0822/Documents/git/bookshop/node_modules/@sap/hana-client/prebuilt/darwinarm64-xcode12/hana-client.node]
 9: 0x11be905ec hana_queue_work(hana_work_t*, Connection*) [/Users/VP-0822/Documents/git/bookshop/node_modules/@sap/hana-client/prebuilt/darwinarm64-xcode12/hana-client.node]
10: 0x11be57cf8 Connection::rollback(Napi::CallbackInfo const&) [/Users/VP-0822/Documents/git/bookshop/node_modules/@sap/hana-client/prebuilt/darwinarm64-xcode12/hana-client.node]
11: 0x11be660ac Napi::InstanceWrap<Connection>::InstanceMethodCallbackWrapper(napi_env__*, napi_callback_info__*)::'lambda'()::operator()() const [/Users/VP-0822/Documents/git/bookshop/node_modules/@sap/hana-client/prebuilt/darwinarm64-xcode12/hana-client.node]
12: 0x11be6602c Napi::InstanceWrap<Connection>::InstanceMethodCallbackWrapper(napi_env__*, napi_callback_info__*) [/Users/VP-0822/Documents/git/bookshop/node_modules/@sap/hana-client/prebuilt/darwinarm64-xcode12/hana-client.node]
13: 0x1026562d0 v8impl::(anonymous namespace)::FunctionCallbackWrapper::Invoke(v8::FunctionCallbackInfo<v8::Value> const&) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
14: 0x10283f714 v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
15: 0x10283f20c v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
16: 0x10283ea98 v8::internal::Builtin_HandleApiCall(int, unsigned long*, v8::internal::Isolate*) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
17: 0x102f7e1cc Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
18: 0x1076cff78 
19: 0x102fc46c0 Builtins_PromiseConstructor [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
20: 0x102f0ef50 Builtins_JSBuiltinsConstructStub [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
21: 0x10773098c 
22: 0x1076ce04c 
23: 0x1076d0f90 
24: 0x102f11d18 Builtins_InterpreterEntryTrampoline [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
25: 0x102f11d18 Builtins_InterpreterEntryTrampoline [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
26: 0x102f11d18 Builtins_InterpreterEntryTrampoline [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
27: 0x102f11d18 Builtins_InterpreterEntryTrampoline [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
28: 0x102f41314 Builtins_AsyncFunctionAwaitResolveClosure [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
29: 0x102fc6178 Builtins_PromiseFulfillReactionJob [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
30: 0x102f336f4 Builtins_RunMicrotasks [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
31: 0x102f0f9e4 Builtins_JSRunMicrotasksEntry [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
32: 0x1028e24cc v8::internal::(anonymous namespace)::Invoke(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
33: 0x1028e2900 v8::internal::(anonymous namespace)::InvokeWithTryCatch(v8::internal::Isolate*, v8::internal::(anonymous namespace)::InvokeParams const&) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
34: 0x1028e29ec v8::internal::Execution::TryRunMicrotasks(v8::internal::Isolate*, v8::internal::MicrotaskQueue*, v8::internal::MaybeHandle<v8::internal::Object>*) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
35: 0x102905628 v8::internal::MicrotaskQueue::RunMicrotasks(v8::internal::Isolate*) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
36: 0x102905ebc v8::internal::MicrotaskQueue::PerformCheckpoint(v8::Isolate*) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
37: 0x1025ddc4c node::InternalCallbackScope::Close() [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
38: 0x1025ddfc8 node::InternalMakeCallback(node::Environment*, v8::Local<v8::Object>, v8::Local<v8::Object>, v8::Local<v8::Function>, int, v8::Local<v8::Value>*, node::async_context) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
39: 0x1025f29ec node::AsyncWrap::MakeCallback(v8::Local<v8::Function>, int, v8::Local<v8::Value>*) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
40: 0x1026caea8 node::worker::MessagePort::OnMessage(node::worker::MessagePort::MessageProcessingMode) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
41: 0x102ef385c uv__async_io [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
42: 0x102f055a8 uv__io_poll [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
43: 0x102ef3cec uv_run [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
44: 0x1025de6d4 node::SpinEventLoop(node::Environment*) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
45: 0x1026c7a90 node::NodeMainInstance::Run(int*, node::Environment*) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
46: 0x1026c7770 node::NodeMainInstance::Run() [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
47: 0x102661e38 node::Start(int, char**) [/Users/VP-0822/.nvm/versions/node/v16.18.1/bin/node]
48: 0x18815be50 start [/usr/lib/dyld]

workerpool options are as below, {"minWorkers":1,"maxWorkers":1,"workerType":"thread","workerThreadOpts":{"resourceLimits":{"maxOldGenerationSizeMb":768}}

The memory usage of main node process during this activity is as below,

2023-02-23_10-35-23 234

  1. Is it normal to see the surge in RSS memory usage to rise from 215MB to 378MB for one worker thread?
  2. Any idea why I see OOM error despite setting --max-old-space-size=8192 for the main node process?

Thanks

josdejong commented 1 year ago

@VP-0822 maybe you can take one of the simple examples provided with the library, for example dedicatedWorker.js, remove the pool..terminate(), and monitor how that behaves on your machine, and see what memory it uses for process vs thread.

It's hard to tell why the OOM error occurs without actually debugging it. An important difference with worker_threads is that they share memory with the main process, where as a child_process has it's own memory. Suppose both your main process and your worker use 6GB of memory. That will work fine with child_process when you have --max-old-space-size=8192, but will be a problem if you use worker threads since the total memory is 12 GB, above the max of 8GB.

The following node.js issue may be relevant too since I see you're configuring both --max-old-space-size=8192 and "maxOldGenerationSizeMb":768: https://github.com/nodejs/node/issues/43991.

VP-0822 commented 1 year ago

@josdejong Thanks for the hint. I raised the issue with the library which is throwing this error and they might fix it soon :)

josdejong commented 1 year ago

👍