search

nodejs / llnode

An lldb plugin for Node.js and V8, which enables inspection of JavaScript states for insights into Node.js processes and their core dumps.
Other
1.15k stars 99 forks source link

[question] empty stacktrace in bt and v8 bt #380

Closed rifler closed 3 years ago

rifler commented 3 years ago

Hi!

I'm a complete newbie with core dump debugging

I have an express.js application running in a cloud Start command is:

node --preserve-symlinks --nouse-idle-notification --max-old-space-size=2048 ./index.js

Sometimes it crashed with long message like:

...some c++ code here...
Aborted (core dumped)\nFATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
...some c++ and js code here...

As I understand it means that process doesn't have enough memory, e.g. memory leak

when I try to run bt or v8 bt in llnode, it shows me empty frames

llnode -c ./my-coredump

(llnode) bt
* thread #1, name = 'node', stop reason = signal SIGABRT
  * frame #0: 0x00007fbfa8456fb7
    frame #1: 0x0000000000a1804c
    frame #2: 0x0000000000b95a7e
    frame #3: 0x0000000000b95df9
    frame #4: 0x0000000000d53075
    frame #5: 0x0000000000d53706
...64 frames...

if I run lldb, it shows me something

lldb `which node` -c ./my-coredump

(lldb) bt
* thread #1, name = 'node', stop reason = signal SIGABRT
  * frame #0: 0x00007fbfa8456fb7
    frame #1: 0x0000000000a1804c node`node::errors::SetEnhanceStackForFatalException(v8::FunctionCallbackInfo<v8::Value> const&) + 284
    frame #2: 0x0000000000b95a7e node`unibrow::Utf8::Encode(char*, unsigned int, int, bool) + 46

Maybe you have advices how I can debug what is going on with fatal error?

Versions: OS - Ubuntu 18.04.3 Node.js, where coredump was generated - v12.20.0 Node.js, where coredump was analyzed - v12.20.0 lldb - 6.0.0 llnode - 3.2.0

addaleax commented 3 years ago

Does v8 bt print something better (in llnode)?

Are you using the exact same binary for inspection that was also used in the crash?

rifler commented 3 years ago

Does v8 bt print something better (in llnode)?

it also prints empty frames. I add LLNODE_DEBUG=true env variable and v8 bt prints some additional message:

CLICK ME 
(llnode) v8 bt
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant off_fp_context, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant off_fp_function, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant off_fp_args, default to -1
[llnode][LoadConstant ../src/constants.cc:104] Failed to load constant off_fp_marker, fallback off_fp_context, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant frametype_ArgumentsAdaptorFrame, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant frametype_EntryFrame, default to -1
[llnode][LoadConstant ../src/constants.cc:104] Failed to load constant frametype_ConstructEntryFrame, fallback frametype_EntryConstructFrame, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant frametype_ExitFrame, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant frametype_InternalFrame, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant frametype_ConstructFrame, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant frametype_JavaScriptFrame, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant frametype_OptimizedFrame, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant frametype_StubFrame, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant SmiTag, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant SmiTagMask, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant SmiShiftSize, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant HeapObjectTag, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant HeapObjectTagMask, default to -1
[llnode][LoadConstant ../src/constants.cc:88] Failed to load constant class_HeapObject__map__Map, default to -1
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61cb74f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61cb7bf is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61cb7df is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d030f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d031f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d037f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d049f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d057f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d05cf is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d060f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d065f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d068f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d070f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d073f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0767 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d07cf is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d082f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d08a7 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0927 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d09a7 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d09e7 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0a5f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0adf is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0b5f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0ba7 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0c1f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0c9f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0d1f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0d9f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0e17 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0e97 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0f17 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0f77 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d0fef is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d106f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d10ef is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d114f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d11c7 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d1247 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d12c7 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d138f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d1407 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d1487 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d1507 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d1587 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d160f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d164f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d1707 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d17a7 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d1817 is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d187f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d196f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d19ff is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d1b2f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d1e0f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d1e3f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d1edf is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d1faf is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d217f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d21cf is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d52ef is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d535f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d547f is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 7ffea61d55ff is not a valid value
[llnode][DoExecute ../src/llnode.cc:82] The value 1ca031f is not a valid value
 * thread #1: tid = 25, 0x00007fbfa8456fb7, name = 'node', stop reason = signal SIGABRT
  * frame #0: 0x00007fbfa8456fb7
    frame #1: 0x0000000000a1804c
    frame #2: 0x0000000000b95a7e
    frame #3: 0x0000000000b95df9
    frame #4: 0x0000000000d53075
    frame #5: 0x0000000000d53706
    frame #6: 0x0000000000d5ffc5
    frame #7: 0x0000000000d60e75
    frame #8: 0x0000000000d6392c
    frame #9: 0x0000000000d317cc
    frame #10: 0x0000000000f72062
    frame #11: 0x0000000000b94cdb
    frame #12: 0x0000000000f742c4
    frame #13: 0x00000000010ab33b
    frame #14: 0x0000000001409219
    frame #15: 0x000000000138d796
    frame #16: 0x0000127f5026976a
    frame #17: 0x0000127f5027809d
    frame #18: 0x000000000138eaa4
    frame #19: 0x0000127f505306eb
    frame #20: 0x0000127f5005d4e9
    frame #21: 0x0000127f5027809d
    frame #22: 0x000000000138eaa4
    frame #23: 0x0000127f505306eb
    frame #24: 0x0000127f501330ff
    frame #25: 0x0000127f5027809d
    frame #26: 0x000000000138eaa4
    frame #27: 0x0000127f505306eb
    frame #28: 0x0000127f5044c3d2
    frame #29: 0x0000127f5027809d
    frame #30: 0x000000000138eaa4
    frame #31: 0x0000127f505306eb
    frame #32: 0x0000127f5002bdaf
    frame #33: 0x0000127f5027809d
    frame #34: 0x000000000138eaa4
    frame #35: 0x0000127f505306eb
    frame #36: 0x000000000138eaa4
    frame #37: 0x0000127f5027809d
    frame #38: 0x000000000138eaa4
    frame #39: 0x0000127f505306eb
    frame #40: 0x0000127f5053e339
    frame #41: 0x0000127f5027809d
    frame #42: 0x000000000138eaa4
    frame #43: 0x0000127f505306eb
    frame #44: 0x0000127f5067b648
    frame #45: 0x0000127f501aefc8
    frame #46: 0x00000000013880fc
    frame #47: 0x000000000138eaa4
    frame #48: 0x0000127f5066aaad
    frame #49: 0x000000000138c01d
    frame #50: 0x000000000138bdf8
    frame #51: 0x0000000000ce7e20
    frame #52: 0x0000000000ce82d8
    frame #53: 0x0000000000ba771b
    frame #54: 0x0000000000a2e681
    frame #55: 0x00000000018940ae
    frame #56: 0x0000000000a30e4f
    frame #57: 0x0000000000af1681
    frame #58: 0x000000000137b239
    frame #59: 0x000000000137b860
    frame #60: 0x0000000001382165
    frame #61: 0x000000000136f8ef
    frame #62: 0x0000000000a5aac6
    frame #63: 0x00000000009e85cc
    frame #64: 0x00007fbfa8439bf7

Are you using the exact same binary for inspection that was also used in the crash?

Yes, I download it from the cloud to my local computer via scp

rifler commented 3 years ago

I fix like this:

1) install lldb-9 sudo apt install --yes lldb-9 liblldb-9 2) use the same version as on production nvm use 12.19.0 3) npm install --lldb_exe=`which lldb-9` -g llnode