search

bnoordhuis / node-heapdump

Make a dump of the V8 heap for later inspection.
Other
2.49k stars 130 forks source link

Generated heapdumps have incorrect retained size? (???) #50

Closed jwalton closed 7 years ago

jwalton commented 9 years ago

I'll prefix this by saying I suspect this isn't actually node-heapdump at fault, but I was wondering if you had some insight into what is going on here? :)

If I generate a heap dump from my app (on Linux, using node 0.10.31), then load this dump into Chrome 40.0.2214.94 (64-bit), in the dev tools every single object has a shallow size which is identical to the retained size:

screen shot 2015-02-04 at 11 38 57 am

If I go to the "Containment" view, I see:

screen shot 2015-02-04 at 11 31 24 am

If I open one of these up, I see the retained size is smaller than the objects inside:

screen shot 2015-02-04 at 11 31 59 am

ChALkeR commented 9 years ago

I have this issue too. Example heapsnapshot from a testcase app: http://oserv.org/bugs/node-heapdump/heapdump-194314979.440694.heapsnapshot.xz

Look at Test or Array objects, for example.

bnoordhuis commented 9 years ago

@jwalton Sorry for the delay, I must have missed the notification email for this issue.

I looked into it and there are at least two issues:

  1. v0.10 heap snapshots are incompatible with recent versions of Chrome Developer Tools. The latter expects fields that are missing in the output from the former. I can perhaps add a script that post-processes the .heapsnapshot file.
  2. v0.12 and io.js also have self size == retained size, which is odd because they are new enough to produce compatible output (io.js is, anyway.)

It's not a bug in Chrome's .heapsnapshot loader; it works fine when you take an in-browser snapshot and save+load it.

I observe that the postorder index step takes a hell of a long time with snapshots created with node-heapdump compared to in-browser snapshots of roughly the same size.

I'm not sure what to make of that. node-heapdump takes the data from V8 pretty much as-is and the version of V8 in io.js is almost identical to the one in Chrome 41.

ChALkeR commented 9 years ago

@vietkute02 mentioned in IRC that a combination of Chrome 37, node-heapdump 0.3.5 (current) and node 0.11.x does not have this bug.

bnoordhuis commented 9 years ago

I observe that the postorder index step takes a hell of a long time with snapshots created with node-heapdump compared to in-browser snapshots of roughly the same size.

I'm not sure what to make of that. node-heapdump takes the data from V8 pretty much as-is and the version of V8 in io.js is almost identical to the one in Chrome 41.

I forgot to mention it here but I did eventually figure out what the issue is - or at least, what I believe the issue is - and it's perhaps also responsible for the self_size == retained_size issue.

Heap snapshots from io.js and node.js processes seem to have much deeper and more cyclic object graphs than in-browser ones. I speculate that DevTools' snapshot loader bails out after it reaches a certain depth but it takes a lot of time and it doesn't figure out what the real retained size is.

DeTeam commented 9 years ago

@bnoordhuis is there any ticket to watch in node/io/chrome or we'd need to report it?

jspavlick commented 9 years ago

I will say that by using Chromium 37 downloaded from here - http://sourceforge.net/projects/osxportableapps/files/Chromium/

I saw self size !== retained size. However, the full size of the heap dump (60MB for me) was still incorrect.

ChALkeR commented 9 years ago

@jspavlick What's your rss/heapTotal/heapUsed? It looks like that the graphs you posted to #62 measure rss and heapdump measures heapUsed. Do Buffers have self size === retained size, btw?

jspavlick commented 9 years ago

Buffers do not have self_size === retained_size in Chromium 37. Not sure about rss/heapTotal/heapUsed, I just took a screenshot of whatever New Relic gave me.

Zarel commented 8 years ago

I'm also having this issue, on iojs 2.3.4 (don't ask).

self size seems correct; it's just retained size that isn't getting calculated.

ChuntaoLu commented 8 years ago

I was facing the same problem of shallow size == retained size, which is solved by enabling "Show advanced heap snapshot properties" check box in the DevTools settings panel.

ChALkeR commented 8 years ago

@ChuntaoLu This works for me for my testcase heapdump linked above.

@bnoordhuis The issue could be closed with documenting that «Show advanced heap snapshot properties» should be enabled in Chrome, I suppose.

bnoordhuis commented 7 years ago

«Show advanced heap snapshot properties»

@ChALkeR Do you know if that option still exists in Chrome 56? I can't find it anymore. FWIW, I haven't received more reports so I suspect this has been fixed upstream.

Refs: https://bugs.chromium.org/p/chromium/issues/detail?id=455374

bnoordhuis commented 7 years ago

I'm fairly sure this has been fixed in Chromium so I'll go ahead and close this out.