Streams introduce flakiness/random results

[jukowski]

I get flacky results when trying to read small files from my hard drive. Sometimes, when access to my harddrive is slow, I get the whole content of the small files. Other times, when content was cached by the OS, the Stream sends "data" events before my code gets to hook into them.

Here is the function I'm using to get the content: {{{

loads file contents into a string

loadVFSFile = (path, _callback) -> async.waterfall([ (callback) -> vfs.readfile(path, {encoding:'utf8'}, callback), (meta, callback) -> data = ''; meta.stream.on("data", (item) -> data += item; ) meta.stream.on("end", () -> callback(null, data); ) return; ], (err, data) -> console.log(err, data); _callback(err, data); ); }}} if I don't use async.waterfall then it works much better but it seems that I don't get any guaranty that it will always work...

[creationix]

Yes, readable streams in node start emitting data events right away. This is actually a real pain point in node. The next major version (node 0.10.x) will have an alternate API for readable streams that doesn't suffer from this difficulty.

From the node docs at http://nodejs.org/api/stream.html

Note that the data will be lost if there is no listener when a Readable Stream emits a 'data' event.

Basically you're created a race condition. You have to attach data listeners in the same tick that vfs gives you the stream in the callback. One thing you can do is use @mikeal's BufferedStream to buffer the events that would be lost. Another solution would be to reorder your logic so that you don't ask for the stream from vfs till you're ready to handle it.