streaming binary files out of lftp

[jeromew]

In order to get a file with ftps, one can either save the file direcly to the filesystem or stream the file into a variable and access it in the callback.

The current implementation in ftps for this is lftp.stdout.on('data', function (res) { data += res; });

This has 2 problems when trying to fetch huge binary files :

• it forces node.js streams into flowing mode (no back-pressure)
• is concats everything to a string which does not work well for binary purposes

If you want, I can try and see how a getFileAsStream could be implemented which would result in a stream. If it works as expected, this would use the module https://www.npmjs.com/package/duplex-child-process which I have been using happily for some time in other circumstances.

[jeromew]

I read the lftp doc and it appears that get always needs to store the content of a file in a local file

get [−E] [−a] [−c] [−e] [−O base] rfile [−o lfile] ... Retrieve the remote file rfile and store it as the local file lfile. If −o is omitted, the file is stored to local file named as base name of rfile. You can get multiple files by specifying multiple instances of rfile (and −o lfile). Does not expand wildcards, use mget for that.

Nevertheless, there is the cat command that says

cat files cat outputs the remote file(s) to stdout. (See also more, zcat and zmore)

so I suppose that it could make sense to have a streaming version of the output of some commands. maybe what would be needed is a more generic execAsStream.

[jeromew]

note: I tested it and it works. instead of exec(cmds, callback) it allows to use var stream = execAsStream(cmds) which is a stream giving the output of the commands. It could be particularly useful to stream the content of a file via cat file.

Tell me what you think about this and I can send a PR adding the execAsStream feature.

[atinux]

I'd love to see the execAsStream feature, you can PR anytime you want!

[atinux]

Available in the 0.6.0 now