Consume stdout in chunks?

[royaldark]

Is it possible to consume the stdout of the command being run line-by-line, or otherwise chunked? I use gulp-run to kick off a long-running (several minutes) script which occasionally writes its progress to stdout. While having a single callback when the command is done is convenient, it would be nice to be able to log those progress messages locally as they come in, before the entire command finishes.

[cbarrick]

The Vinyl file pushed down the gulp pipeline is just a thin wrapper of the command's stdout stream. This means you (or other gulp-plugins) can read stdout in real time, allowing you to buffer it however you want. Below is an example showing that you can call .read() on the virtual file to read stdout. That being said, other plugins my buffer the contents before sending downstream.

If your main concern is logging to a file, you might consider using a tee(1) in your shell command to tee stdout to a file, or you could use a logger plugin in your gulp pipeline. Both approaches should be able to log stdout in real time.

The callback argument to exec is really just a convenience. The primary interface with the standard I/O is using Vinyl files.

// (import gulp-run like always)
> run = require('./')
{ [Function: run] Command: [Function: Command] }

// A call to run gives us a gulp stream, i.e. a stream of Vinyl objects.
// In our case, for each invocation of the shell command, one Vinyl object
// is pushed down the stream representing the stdout of the sub-shell.
// The command we're using echos "hello" to stdout, then waits for 30s
// before echoing "world".
> stream = run('echo -n hello && sleep 30 && echo world')
{...}

// A call to exec() causes the shell command to be invoked. This pushes a Vinyl
// object into the stream that represents this invocation's stdout.
// (The stream itself is returned, allowing you to chain method calls)
> stream.exec()
{...}

// To do anything with the output, we need to get the Vinyl object out of the
// stream. Normally we just pipe it along to the next plugin, but we consume it
// here to examine how Vinyl objects work. Take a look:
> file = stream.read()
<File "echo" <PassThroughStream>>

// Vinyl objects are "virtual files". In this case, our file is named "echo"
// after the first word in the command, and our file's contents is a
// PassThroughStream hooked up to stdout on the underlying subshell that
// is processing the command. We can read some data out of that stream:
> file.contents.read()
<Buffer 68 65 6c 6c 6f 0a>

// The results of our read are the bytes written out thus far by the command.
// Reading from the contents also consumes it, so future reads return `null`.
> file.contents.read()
null

// Eagle eyed observers would note that our first two reads only consumed the
// string "hello". That's because they were executed in the 30s sleep between
// the command's writing of "hello" and "world". After 30s, we can read the rest
// of the output from the shell command.
> file.contents.read()
<Buffer 77 6f 72 6c 64 0a>

I'm closing this issue, as a buffered interface is a specific non-goal of this project, but I hope the example is helpful and even more that you find a solution to your problem.

[royaldark]

Thank you, your explanation was very helpful!