search

Shiranuit / node-segfault-handler

A C++ Node.js module that helps gathering informations on segmentation fault
BSD 3-Clause "New" or "Revised" License
14 stars 1 forks source link

Tips on understanding a stacktrace (not related to lib) #31

Open jakew009 opened 1 year ago

jakew009 commented 1 year ago

Hello

Thanks for a super useful package!

I wonder if you can give us any clues as to how we go about debugging the below stacktrace.

This is the only output we are getting from the segfault handler:

root@ifu-20:/data/crashlogs# cat segfault_1677286402019.log
=========== Caught a Segmentation Fault [pid=417] ===========
-----[ Native Stacktraces ]-----
[pc=0x00000000b43269a0, sp=0x00000000bec5bb68] in segfault_handler(int)+0x48
[pc=0x00000000b6680061, sp=0x00000000bec5bb78] in sigset+0xd8
[pc=0x000000003ca490c8, sp=0x00000000bec5be74] in +0xd8

---[ V8 JavaScript Stacktraces ]---
============================================================

Have you got any ideas about where we go from here to debug it? We only see it sporadically, sometimes it will happen a few times in a row and then not happen again for weeks. We are going to start disabling node modules that have native code and see if we can narrow it down that way.

Shiranuit commented 1 year ago

Sorry, this is the first time I see this happening, it seems that there is no stacktrace to unwind, which seems weird. It could be a bug or maybe there is nothing in the stacktrace but this is weird since there should be at least some functions from NodeJS.

jakew009 commented 1 year ago

Hi, thanks for such a fast reply!

The reason for installing the plugin is we are trying to track down a sporadic segfault on an embedded system. It usually segfaults a couple of times after starting (1-3 times) and then seems to run fine for a few days.

We do sometimes get more sensible stacktraces, so i think the actual library is working. For instance, we have seen this stack trace, but I don't believe it actually give us any clues as to the source of the crash. Most of the time we get the previous stack trace (with very few details).

root@ifu-20:/data/crashlogs# cat segfault_1677438927471.log
=========== Caught a Segmentation Fault [pid=16179] ===========
-----[ Native Stacktraces ]-----
[pc=0x00000000b432e9a0, sp=0x00000000be871c10] in segfault_handler(int)+0x48
[pc=0x00000000b6688061, sp=0x00000000be871c20] in sigset+0xd8
[pc=0x0000000000bd00f2, sp=0x00000000be871f18] in v8::internal::Heap::MergeAllocationSitePretenuringFeedback(std::unordered_map<v8::internal::AllocationSite, unsigned int, v8::internal::Object::Hasher, std::equal_to<v8::internal::AllocationSite>, std::allocator<std::pair<v8::internal::AllocationSite const, unsigned int> > > const&)+0x2d

---[ V8 JavaScript Stacktraces ]---
at handleWriteReq (internal/stream_base_commons.js:63:21)
at writeGeneric (internal/stream_base_commons.js:145:15)
at Socket._writeGeneric (net.js:787:11)
at Socket._write (net.js:799:8)
at doWrite (_stream_writable.js:403:12)
at writeOrBuffer (_stream_writable.js:387:5)
at Writable.write (_stream_writable.js:318:11)
at log (/usr/lib/node/ifu-node/node_modules/winston/lib/winston/transports/console.js:79:23)
at _write (/usr/lib/node/ifu-node/node_modules/winston-transport/index.js:103:17)
at doWrite (/usr/lib/node/ifu-node/node_modules/winston-transport/node_modules/readable-stream/lib/_stream_writable.js:409:139)
at writeOrBuffer (/usr/lib/node/ifu-node/node_modules/winston-transport/node_modules/readable-stream/lib/_stream_writable.js:398:5)
at Writable.write (/usr/lib/node/ifu-node/node_modules/winston-transport/node_modules/readable-stream/lib/_stream_writable.js:307:11)
at ondata (/usr/lib/node/ifu-node/node_modules/winston/node_modules/readable-stream/lib/_stream_readable.js:681:20)
at emit (events.js:314:20)
at EventEmitter.emit (domain.js:483:12)
at addChunk (/usr/lib/node/ifu-node/node_modules/winston/node_modules/readable-stream/lib/_stream_readable.js:298:12)
at readableAddChunk (/usr/lib/node/ifu-node/node_modules/winston/node_modules/readable-stream/lib/_stream_readable.js:280:11)
at Readable.push (/usr/lib/node/ifu-node/node_modules/winston/node_modules/readable-stream/lib/_stream_readable.js:241:10)
at Transform.push (/usr/lib/node/ifu-node/node_modules/winston/node_modules/readable-stream/lib/_stream_transform.js:139:32)
at _transform (/usr/lib/node/ifu-node/node_modules/winston/lib/winston/logger.js:313:12)
at Transform._read (/usr/lib/node/ifu-node/node_modules/winston/node_modules/readable-stream/lib/_stream_transform.js:177:10)
at Transform._write (/usr/lib/node/ifu-node/node_modules/winston/node_modules/readable-stream/lib/_stream_transform.js:164:83)
at doWrite (/usr/lib/node/ifu-node/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:409:139)
at writeOrBuffer (/usr/lib/node/ifu-node/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:398:5)
at Writable.write (/usr/lib/node/ifu-node/node_modules/winston/node_modules/readable-stream/lib/_stream_writable.js:307:11)
at value (/usr/lib/node/ifu-node/node_modules/winston/lib/winston/logger.js:67:18)
at DerivedLogger.<computed> (/usr/lib/node/ifu-node/node_modules/winston/lib/winston/create-logger.js:81:14)
at storeCurrentRead (/usr/lib/node/ifu-node/dist/infra/power/sensors/PowerSystemSensor.js:56:16)
at monitor (/usr/lib/node/ifu-node/dist/infra/power/sensors/PowerSystemSensor.js:34:18)
============================================================

Q: Are you registering the segfault handler before importing any other node_modules ? A: We were not, but i have just changed the code to do so. The first line of the entry point to the app is now

import segfaultHandler from 'node-segfault-handler'
segfaultHandler.registerHandler(`/data/crashlogs/segfault_${Date.now()}.log`)

// lots of other imports
import { baseLogger } from './logger'
etc

Q: Which OS are you using ? A: It's actually a custom Yocto build running on a Raspberry Pi.

Q: Can you tell me which version of NodeJS you're using ? A: 12.22.12, we are working on upgrading to see if the problem goes away with a newer version.

Q: How is the libunwind installed on your system ? A: We installed libunwind as part of the Yocto build process.

Q:Which version of the libunwind is installed ? A: 1.3.1

As an aside, do you do any consulting?

Shiranuit commented 1 year ago

Hi, sorry for the late reply, had a lot of work.

I was wondering if this issue was linked to Yocto, because while doing some research I saw that there was many issue related to Node and Yocto involving segmentation fault, I also saw that there was some issue with libunwind and Yocto so this might be why the stacktrace is not unwinded properly.

As an aside, do you do any consulting?

Unfortunately I'm not doing any consulting, I dont have much time for it.