running GC appears to interfere with jsconstructor

davepacheco commented 6 years ago

After making some changes for #27, I found tst.postmortem_details.js failing sometimes as follows:

$ node test/standalone/tst.postmortem_details.js
mdb stderr: mdb: unable to read object map

mdb stderr: mdb: unable to read object map

mdb stderr: mdb: unable to read object map

verifying test: jsconstructor using verifyConstructor
verifying test: nodebuffer using verifyNodebuffer
verifying test: nodebuffer contents using verifyBufferContents
verifying test: v8internal using verifyV8internal
verifying test: sliced buffer using verifySlicedBufferConstructor

assert.js:86
  throw new assert.AssertionError({
        ^
AssertionError: [] deepEqual [ 'NativeBuffer' ]
    at Array.verifySlicedBufferConstructor (/home/dap/mdb_v8/test/standalone/tst.postmortem_details.js:165:11)
    at ChildProcess.<anonymous> (/home/dap/mdb_v8/test/standalone/tst.postmortem_details.js:101:26)
    at ChildProcess.emit (events.js:110:17)
    at Process.ChildProcess._handle.onexit (child_process.js:1078:12)

It's the part of the test that locates the SlicedBuffer it creates and uses jsconstructor to print its constructor. jsconstructor is failing with mdb: unable to read object map.

There are a couple of interesting data points:

when I apply an mdb_v8.so compiled from master to the same core file left by the above failing test, the same jsconstructor command fails
when I use a copy of the mdb_v8 source from master, modify the test to leave its core file, and use my updated mdb_v8.so on the core file, the jsconstructor command succeeds
the test program sometimes succeeds even in my dev branch (probably about 50% of the time on 64-bit Node v0.12.17. I'm not sure I've seen the failure at all on other builds, though I've mostly been using 0.10.)
if I add a line to the test that prints the constructor name, the test reliably works.

So it seems as though the core file being generated is sometimes different based on some V8 implementation choice, not that there's new breakage in my dev version of mdb_v8.so. My conclusion is that there's some case that jsconstructor doesn't handle. This ticket covers handling that case. I will upload the test case core file to thoth.

davepacheco commented 6 years ago

Test case core file:

$ mdb -S /var/tmp/node.74904
Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ]
> ::load /home/dap/mdb_v8_pristine/build/amd64/mdb_v8.so
mdb_v8 version: 1.2.1 (dev)
V8 version: 3.28.71.19
Autoconfigured V8 support from target
C++ symbol demangling enabled
> 2bde4c0047d9::jsconstructor
mdb: unable to read object map

This is thoth dump 1133bc4713168a309ca5ad64a3818514.

davepacheco commented 6 years ago

When I looked at this last week, I missed that we were in the middle of a garbage collection when this core file was created:

> ::jsstack                           
native: libc.so.1`mmap+0xa
native: v8::internal::MemoryAllocator::AllocateAlignedMemory+0x49
native: v8::internal::MemoryAllocator::AllocateChunk+0x84
native: v8::internal::PagedSpace::Expand+0x70
native: v8::internal::PagedSpace::SlowAllocateRaw+0x4a
native: _ZN2v88internal17ScavengingVisitorILNS0_13MarksHandlingE1ELNS0_1...
native: _ZN2v88internal17ScavengingVisitorILNS0_13MarksHandlingE1ELNS0_1...
native: v8::internal::StoreBuffer::IteratePointersInStoreBuffer+0x7a
native: v8::internal::StoreBuffer::IteratePointersToNewSpace+0x49
native: v8::internal::Heap::Scavenge+0x2f1
native: v8::internal::Heap::PerformGarbageCollection+0xef
native: v8::internal::Heap::CollectGarbage+0x14f
native: v8::internal::Factory::NewFixedArray+0x7a
native: v8::internal::JSObject::MigrateFastToFast+0x13a
native: v8::internal::JSObject::MigrateToMap+0x8e
native: v8::internal::LookupIterator::TransitionToDataProperty+0xa6
native: v8::internal::Object::AddDataProperty+0x109
native: v8::internal::Object::SetProperty+0xcb
native: v8::internal::Object::SetProperty+0xe4
native: v8::internal::StoreIC::Store+0x13a
native: v8::internal::StoreIC_Miss+0x9f
        (1 internal frame elided)
js:     <anonymous> (as EventEmitter.init)
js:     EventEmitter
js:     Stream
js:     Readable
js:     Duplex
js:     Socket
        (1 internal frame elided)
js:     createSocket
js:     <anonymous> (as <anon>)
        (1 internal frame elided)
js:     forEach
        (1 internal frame elided)
js:     <anonymous> (as ChildProcess.spawn)
js:     <anonymous> (as exports.spawn)
        (1 internal frame elided)
js:     <anonymous> (as <anon>)
        (1 internal frame elided)
js:     <anonymous> (as Module._compile)
js:     <anonymous> (as Module._extensions..js)
js:     <anonymous> (as Module.load)
js:     <anonymous> (as Module._load)
js:     <anonymous> (as Module.runMain)
js:     startup
js:     <anonymous> (as <anon>)
        (1 internal frame elided)
        (1 internal frame elided)
native: _ZN2v88internalL6InvokeEbNS0_6HandleINS0_10JSFunctionEEENS1_INS0...
native: v8::internal::Execution::Call+0xb3
native: v8::Function::Call+0xc2
native: node::LoadEnvironment+0x1fe
native: node::Start+0x157
native: _start+0x6c

That probably explains with mdb_v8 isn't able to do anything with it. This is not a simple problem to solve, since during GC, V8 mucks with address values, and I think we can't tell what's a pointer and what's an integer.

TritonDataCenter / mdb_v8

running GC appears to interfere with jsconstructor #103