Memory usage with fs.createReadStream()

[julien-f]

require('trace')

var stream = require('fs').createReadStream('huge file here')
stream.resume()

It seems that the entire file is loaded in memory.

[AndreasMadsen]

There doesn't appear to be any memory leaks, but there is an odd behaviour in the RSS part which usually comes from new Buffer allocations. I have no idea what it is about, trace doesn't change the behaviour of node in any of those areas.

baseline

with trace

[julien-f]

Side question: which tool are you using to analyse memory usage in Node?

[AndreasMadsen]

I just wrote something simple: https://github.com/AndreasMadsen/trace/commit/7e5496aca15b821a5e3306f58cd7d62255331c3e

maybe I will make a real module some day :D

[AndreasMadsen]

7b91e4a5487bf0b998b69465b4528747fc2ff510 should fix this. If you try it out (please do) be aware that you will need at least node version 5.2.0 for now. However I have been promised that the necessary node features will land soon in 4.x.

[AndreasMadsen]

I don't get why this is an issue. I suspect it might be a v8 bug in the gc, 7b91e4a5487bf0b998b69465b4528747fc2ff510 basically just moved the stack object into an id-map and does the memory management itself. For future reference I also tried using a WeekMap, so I din't have to use the gc destroy event, there was an improvement but not enough.

WeakMap using handle object

Map using uid and destructor event

[AndreasMadsen]

Thanks for creating the issue. I really appreciate the time you use on trace and stack-chain :+1:

[julien-f]

Glad to help.

Thanks for your work :)